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16  Abstr0ct_a*  rep0rt  describes  the  results  of  an  experiment  for  evaluating  the 

controller/computer  interface  in  an  ARTS  III/M&S  system  modified  for  use  with  a 
simulated  digital  data  link  and  a voice  link  utilizing  a computer-generated  voice 
system.  A modified  ARTS  III  M&S  system  at  the  National  Aviation  Facilities  Experi- 
mental Center  (NAFEC)  provided  the  means  for  determining  which  of  three  candidate 
control  and  display  modes  was  the  most  suitable  for  the  display  and  dispatch  of  com- 
puter-generated M&S  commands  in  a mixed  voice/digital  communications  environment . 

The  three  modes  tested  were  Control-by-Approval/Full  Data  Block  (CBA/FDB) , ControA^\\ 
by-Approval/Tabular  List  (CBA/TAB) , and  Control-by-Exception/Full  Data  Block  (CBE/Vm' 
In  Control-by-Approval,  the  controller  must  approve  each  command;  in  Control-by- 
Exception,  the  controller  must  intervene  to  disapprove  commands  that  are  otherwise 
automatically  dispatched.  The  three  modes  were  tested  by  NAFEC  air  traffic  control 
specialists  in  an  M&S  scenario  simulating  south  arrivals,  single  runway  only,  at 
Denver  Stapleton  International  Airport.  Figures  of  merit  for  evaluating  the  three 
modes  included  subjective  data  in  the  form  of  questionnaires  from  participating 
controllers  and  objective  data  such  as  message  transaction  time,  command  initiation 
delay,  service  time,  and  instantaneous  aircraft  load.  The  results  of  66  hours  of 
testing  with  six  air  traffic  controllers  show  that  the  fully  automated  CBE/FDB 
control  mode  is  preferred  because  it  possesses  the  best  workload,  capacity  and  stress 
characteristics.  Although  the  CBA/TAB  and  CBA/FDB  modes  tend  to  exhibit  shorter 
communications  delays,  the  former,  with  its  list  display,  diverts  attention;  and  the 
latter,  with  its  trackball,  increases  workload. 
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PREFACE 


This  report  contains  the  results  of  an  evaluation  of  the 
controller/computer  interface  studied  in  the  Data  Link  Experiments 
Program  managed  by  the  Transportation  Systems  Center  and 
sponsored  by  the  FAA/SRDS.  The  primary  objective  of  this 
evaluation  effort  was>  to  test  three  candidate  control  and 
display  modes  to  be  used  by  air  traffic  controllers  in  an 
ARTS  IIl/M^S  System  modified  for  operation  in  a simulated 
mixed  voice/digital  communications  environment  using  a scenario 
based  on  Denver  Stapleton  International  Airport.  The  evaluation 
system  was  implemented  and  tested  at  the  NAFF.C  TATF  (Terminal 
Automation  Test  Facility). 
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EXECUTIVE  SUMMAPY 


S.l  BACKGROUND 

The  Department  of  Transportation/Transportation  Systems  Center 
(TSC)  under  the  sponsorship  of  the  FAA/SRDS  has  been  engaged  in  a 
Data  Link  Experiments  Program  aimed  at  evaluating:  (1)  the 

controller/computer  interface  in  a digital  data  link  communica- 
tions environment;  (2)  characteristics  of  a VHF  data  link  between 
air  and  ground  stations;  ard  (3)  cockpit  I/O  devices  that  take 
advantage  of  the  automatic  capability  of  a data  link  channel.  This 
report  is  concerned  only  with  the  controller/computer  interface 
effort,  designated  Phase  2B,  which  was  preceded  by  earlier  prelimi- 
nary work  under  Phase  2A  at  the  NAFEC  Digital/Simulation  Facility. 
The  Phase  2B  experiment  was  conducted  at  the  NAFEC  Terminal  Auto- 
mation Test  Facility  (TATF)  with  an  ARTS  III  M§S  system  modified  for 
operation  with  a simulated  data  link  and  a voice  link  utilizing 
computer-generated  voice.  The  primary  objective  of  the  Phase  2B 
experiment  was  to  ascertain  which  of  the  following  three 
candidate  control  modes  identified  in  the  Phase  2A  experiment 
is  the  most  acceptable  to  air  traffic  controllers  for  the  display 
and  dispatch  of  M$S  commands: 

1.  Control-by-Approval/Full  Data  Block  (CBA/FDB) 

2.  Control -by-Approval/Tabular  List  (CBA/TAB) 

3.  Control-by-Exception/Full  Data  Block  (CBE/FDB) 

In  the  two  CBA  modes,  the  controller  must  approve  each  command; 
in  the  CBE/FDB  mode,  the  controller  intervenes  only  to  disapprove 
commands  which  would  otherwise  be  automatically  dispatched.  In 
the  CBA/FDB  and  CBE/FDB  modes,  up  to  three  commands  can  be  dis- 
played simultaneously  in  the  third  line  of  the  ARTS  data  block; 
in  the  CBA/TAB  mode,  commands  are  displayed  one  per  line  in  a 
list. 
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S. 2 PROCEDURE 


Each  of  six  NAFEC  air  traffic  control  specialists  tested  each 
of  the  three  data  link  control  modes  against  three  traffic  mixes 
consisting  of  20%,  501,  and  80%  data  link  aircraft  in  a scenario 
based  on  Denver  Stapleton  International  Airport.  In  addition, 
each  controller  tested  the  two  control -by-approval  modes  without 
the  simulated  data  link  and  voice  response  system  so  that 
potential  improvements  obtainable  via  the  data  link  modes 
could  be  compared  with  "pure"  voice  mode  data. 

A total  of  66  1-hour  test  runs  were  conducted  during  which 
both  controllers  and  pilots  were  watched  by  test  observers.  Each 
controller  received  a 1-hour  training  run  immediately  before 
testing  a different  control  mode.  The  schedule  for  each 
controller's  test  run  was  carefully  arranged  in  a master  test 
matrix  :o  balance  out  sequence  effects.  The  data  from  each  test 
run  consisted  of  controller  questionnaire  responses  and  data  re- 
duced and  processed  by  extraction  programs  which  measured  the 
following : 

1.  Message  transaction  time,  command  initiation  delay, 
instantaneous  aircraft  load,  etc. 

2.  Counts  of  the  number  of  voice  and  data  link  commands, 
extemporaneous  communications,  handoffs,  tag  reposition- 
ings, etc. 

3.  Aircraft  position  for  subsequent  input  to  conflict 
detection  programs. 

The  traf  "ic  samples,  consisting  of  approximately  30  aircraft,  were 
designed  to  subject  the  controller  to  a linear  increase  in  in- 
stantaneous aircraft  load  from  0 to  9 aircraft  during  the  1-hour 
test  period.  This  facilitated  the  continuous  measurement  of  the 
controller's  communication  delays,  which  were  expected  to  increase 
until  his  workload  limit  was  reached  and  then  increase  rapidly 
thereafter. 
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S . 3 RESULTS 

On  the  basis  of  controller  preference  as  manifested  in  the 
questionnaires,  the  "hands  off"  CBE/FDB  mode  was  ranked  first, 
CBA/TAB , second,  and  CBA/ FDB.  1 ast . CBE/FDB  had  the  longest  mes- 
sage transaction  time  (MTT)  of  the  three  modes  but  was  ranked 
first  by  the  controllers,  despite  some  concern  for  blunder 
potential,  because  it  possessed  the  best  workload,  capacity,  and 
stress  characteristics.  Longer  MTT's  are  inherent  in  CBF./FDR 
because  command  initiation  delay  (CID)  is  prolonged  in  order  to 
provide  command  disapproval  time.  CBA/TAB,  with  its  single-key 
command  dispatch  had  the  shortest  MTT,  but  was  ranked  second  by 
the  controllers  because  they  feel  operating  from  a command  list 
tends  to  divide  their  attention,  adversely  affecting  workload  and 
capacity.  CBA/FDB  had  the  second  best  MTT  but  it  was  consistently 
rated  as  unfavorable  in  workload,  capacity,  stress,  and  blunder 
potential  because  of  the  necessity  for  continuous  trackball  slewing 
action . 

S . 4 CONCLUSIONS 

1.  The  CBF/FDB  mode  is  the  most  favored  by  the  controllers 
despite  blunder  potential,  but  some  inherent  limitations 
result  in  longer  message  transaction  times. 

2.  As  was  concluded  in  the  Phase  2A  experiment,  a CBA/FDB 
mode  in  which  a single  command  is  displayed  at  one  time 
in  one  data  block  for  dispatch  with  only  a single 
keyboard  entry  is  again  indicated  as  a potentially 
viable  mode  in  a future  system.  This  mode  would 
combine  the  advantages  of  both  the  CBA/FDB  and  the 
CBA/TAB  modes  as  currently  implemented. 

3.  The  Phase  2B  computer-generated  voice  system  was  not 
favorably  regarded  because  of  the  talking  rate  and 
inability  to  repeat  commands. 
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S.5  RECOMMENDATIONS 

Further  studies  aimed  at  determining  the  optimum  means  for 
interfacing  the  controller  and  the  computer  in  a data  link  com- 
munications environment  should  be  conducted  in  the  following  areas: 

1.  Additional  testing  of  the  Control -by- Except ion/Ful 1 
Data  Block  mode  of  operation  should  be  conducted  with  an 
on-line  controller  selectable  disapproval  time.  This 
will  allow  a controller  to  decrease  MTT  if  he  feels 
disapproval  time  need  not  be  as  long  (6  seconds)  as  it 
was  in  this  study. 

2.  On  the  basis  of  the  findings  of  both  Cont rol ler/Computer 
Interface  experiments  (Phase  2A  and  2B)  it  is  recommended 
that  a CBA/FDB  mode  (in  which  a single  command  is  dis- 
played at  one  time  in  an  aircraft  data  block  for  dis- 
patch with  only  a single  keyboard  entry)  be  incorporated 
in  a future  system. 

3.  Considering  the  advantage  of  computer-generated  voice 
in  providing  commonality  of  command  dispatch  in  a mixed 
voice/digital  communications  environment,  it  is  recom- 
mended that  future  research  be  devoted  to  improving  the 
talking  rate  and  in  developing  a command  repeat 

capab il i ty . 
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1,  INTRODUCTION 


The  Department  of  Transportation/Transportation  fycre™-  renter 
(TSC)  , under  the  sponsorship  of  the  FAA/SRDS,  has  been  engaged  in  a 
Data  Link  Experiments  Program  aimed  at  evaluating  (1)  the 
cont rol 1 er/comput er  interface  in  a digital  data  link  communica- 
tions environment,  (2)  characteristics  of  a VHF  data  link  between 
air  and  ground  stations,  and  (3)  cockpit  I/O  devices  that  take 
advantage  of  the  automatic  capability  of  a data  link  channel.  This 
report  is  concerned  with  the  evaluation  of  the  controller/computer 
interface  portion  of  the  program  designated  as  Phase  2B.  The 
Phase  2B  evaluation  system  was  implemented  at  the  NAFEC  Terminal 
Automation  Test  Facility  and  consisted  of  the  ARTS  III/Mf,S  system 
modified  for  operation  with  a simulated  data  link  and  a voice  link 
utilizing  a voice  response  system  (VRS) . The  evaluation  system 
was  operated  in  three  controller  modes  for  the  display  and  dis- 
patch of  computer-generated  M§S  commands  and  the  display  of  air- 
craft responses.  The  three  modes  were  Control-by-Approval/Full 
Data  Block,  Control-by-Approval/Tabular  List,  and  Control-by- 
Except ion/Ful 1 Data  Block.  In  Control -by-Approval , the  controller 
must  approve  each  command;  in  Control -by-Exception , the  controller 
intervenes  only  to  disapprove  commands  which  would  otherwise  be 
automatically  dispatched.  In  the  Full  Data  Block,  up  to  three 
commands  are  displayed  simultaneously  in  the  third  line  of  the 

ARTS  data  block;  in  the  Tabular  List,  they  are  displayed  one  per  J 

line  in  a list. 


It  should  be  pointed  out  that  the  selection  of  the  three  data 
link  control  modes  for  evaluation  was  based  on  the  results  of  a 
previous  controller/computer  interface  experiment  conducted  at  the 
NAFEC  Digital  Simulation  Facility  (DSF) . In  the  DSF  experiment, 
an  ARTS  III/Mf,S  system  operating  within  a mixed  voice/digital 
communications  environment  was  simulated  and  evaluated  by  air 
traffic  controllers  for  the  purpose  of  identifying  the  most 
promising  of  seven  control  modes  to  be  subsequently  tested  in  a 
real  ARTS  III/M5S  system  at  the  NAFEC  TATF. 
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This  report  describes  the  results  of  a program  of  evaluation 
using  air  traffic  controllers  and  simulation  pilots  "flying" 
aircraft  created  by  the  ARTS  internal  target  generator. 

Each  controller  evaluated  each  of  the  three  control  modes  in  an 
M§S  scenario  simulating  south  arrivals,  single  runway  only,  at 
Denver  Stapleton  International  Airport.  In  addition  to  operating 
the  modified  ARTS  III  M5S  system  with  the  three  data  link  control 
modes,  each  controller  operated  the  system  in  the  Control-by- 
Approval/Full  Data  Block  and  Control-by-Approval/Tabular  List 
modes  without  benefit  of  a simulated  data  link  channel  or 
voice  response  system.  In  this  manner,  performance  data  were 
obtained  for  the  "pure"  voice  modes  against  which  potential  im- 
provements obtainable  via  the  data  link  modes  could  be  compared. 
Three  different  traffic  samples,  each  having  a different  ratio  of 
data  link  to  voice  link  aircraft,  were  used  in  the  evaluation  of 
each  of  the  data  link  modes;  a different  traffic  sample  was  used 
for  each  of  the  two  voice  modes. 


2.  TEST  OBJECTIVES 

The  primary  objective  of  this  test  program  was  to  evaluate 
three  candidate  control  and  display  modes  to  be  used  by  air  traffic 
controllers  in  an  ARTS  III/M^S  system  modified  for  operation  in  a 
mixed  voice/digital  communications  environment  using  a simulation 
based  on  Denver  Stapleton  Internation  Airport.  The  ARTS  IIl/MfjS 
system  was  used  in  this  evaluation  program  because  it  provided 
a high  volume  of  the  automatically  generated  ATC  messages  that 
comprise  a large  portion  of  the  controller's  total  communications 
workload.  The  high  message  rates  obtainable  with  the  NUiS  system 
permitted  a meaningful  comparison  of  controller  reaction  to 
each  of  the  three  control  modes  with  respect  to  the  display, 
dispatch,  and  receipt  of  data  link  messages.  Analysis  of  the  test 
results  provided  subjective  and  objective  data  pertaining  to  the 
following : 

a.  The  controller's  ability  to  deal  effectively  with 
increasing  numbers  of  data  link  aircraft  in  traffic 
situations  involving  a high  volume  of  control  messages. 

b.  The  control  mode  which  is  most  acceptable  to  the  con- 
troller and  enhances  his  traffic  handling  capacity 
without  degrading  his  ability  to  evaluate  commands. 


V 


ASSUMPTIONS  AND  CONSTRAINTS 

As  previously  stated,  this  test  program  was  the  second  stage 
of  a two-stage  effort  involving  TSC  and  NAFEC.  The  first  stage 
was  conducted  at  the  NAFEC  DSF  using  a simplified  model  of  the 
metering  and  spacing  traffic  pattern  in  a simulated  ARTS  III 
system.  The  assumptions  and  constraints  governing  the  conduct 
of  this,  the  second  stage,  were  as  follows: 

a.  The  actual  ARTS  III  prototype  M§S  system  (November  1973 
version),  modified  to  provide  data  link  display  identifiers 
and  keyboard  functions,  would  be  used  at  the  NAFEC  Terminal 
Automation  Test  Facility  (TATF) . 

b.  The  Denver  Stapleton  Airspace  Geometry  used  in  previous 
M§S  development  testing  would  be  used,  but  terminal  area 
traffic  would  be  limited  to  south  arrivals  through  two 
feeder  fixes  (Shawnee  and  Elizabeth)  onto  a single  run- 
way (26L)  handled  by  one  air  traffic  controller. 

c.  The  control  modes  to  be  tested  would  be  the  few  most 
promising  ones  identified  in  the  first  stage  of  testing 
at  the  DSF. 

Each  of  the  above  is  discussed  in  more  detail  below. 

3.1  ARTS  III  M§S  SYSTEM  MODIFIED  FOR  DATA  LINK 

3.1.1  lias ic  ARTS  III  MSS  System 

The  existing  ARTS  III  M§S  system  replaces  the  controller's 
mentally  calculated  heading,  altitude,  and  speed  commands  with 
those  that  are  automatically  generated  by  a computer  algorithm 
that  regulates  traffic  flow  in  the  entire  terminal  area.  Commands 
generated  by  the  algorithm  are  displayed  on  the  ARTS  III  Data  Entry 
and  Display  Subsystem  (DEDS)  in  either  the  full  data  block  or  in  a 
tabular  list  for  voice  delivery  from  controller  to  pilot.  The  two 
methods  of  displaying  commands  are  described  below  and  depicted  in 
Figures  1 and  2. 
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(FDB  without  M$S  Command  in  third  line) 
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A 

EA123 

(Fly  heading  200°) 

160  25 
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A 

EA12  3 

(Descend  and  maintain  12,000  feet) 

160  25 

;A120;. 

A 

EA123 

(Fly  heading  180°,  descend  and  maintain 

160  2 5 

10,000  feet) 
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NOTE: 

The  second  line  of  the  data  block  is  ARTS  III  display  of 
altitude  and  ground  speed.  A is  the  controller  symbol  and 
denotes  aircraft  position.  Blinking  commands  are  encircled 
with  dots. 

Figure  1.  Mf)S 

Command  Display  in  Full  Data  Block 

EA12  3 

SI  20 

(Reduce  Speed  120  Knots) 

AA155 

HI  8 0 

(Fly  heading  180°) 

AL924 

A080 

(Descend  and  maintain  8000  feet) 

TW331 

H270 

(Fly  heading  270°) 
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3. 1.1.1  Full  Data  Block  - Combinations  of  MSS  heading,  altitude, 
and  speed  commands  are  displayed  blinking  in  the  third  line  of  the 
ARTS  III  data  block.  The  commands  appear  blinking  for  an  interval 
during  which  they  can  be  voiced  by  the  controller  for  exgtj-nt  inn  — . 
by  the  aircraft.  After  the  blink  interval  has  elapsed,  the  com- 
mands stop  blinking,  briefly  remain  on  the  display,  and  are  then 
automatically  erased.  The  controller  can  use  the  trackball  with 

a special  function  key  to  erase  the  command  after  voicing  it  to 
the  pilot.  By  so  doing,  he  can  better  distinguish  pending  com- 
mands from  those  that  he  had  voiced. 

3. 1.1. 2 Tabular  List  - Heading,  altitude,  or  speed  commands  are 
displayed  one  at  a time,  in  a separate  list.  The  command  timing 
and  voice  delivery  are  similar  to  those  used  in  the  full  data 
block,  except  that  the  controller  can  erase  only  the  topmost  com- 
mand in  the  list.  In  such  instances,  all  other  commdands  would 
be  moved  up  one  line. 

3.1.2  Modifications  for  Data  Link 

The  existing  ARTS  III  MSS  system  display  presentation  and 
operational  features  were  carefully  evaluated  in  order  to  define 
the  means  by  which  the  system  would  have  to  be  modified  (Reference 
1)  to  make  addition  of  a data  link  capability  to  the  controller's 
MSS  communications  environment  operationally  feasible  and  acceptable. 
In  determining  what  modifications  were  required,  this  program  bene- 
fited significantly  from  the  large  body  of  original  work  done  in 
this  area  in  the  design,  implementation,  and  testing  of  the  DSF 
experimental  system. 

The  principal  areas  of  modification  identified  included  the 
following : 

a.  Designation  of  data  link  equipped  aircraft 

b.  Indication  of  pilot  responses  to  MSS  commands 

c.  Incorporation  of  function  keys  for  command  selection 

d.  Incorporation  of  computer-generated  voice  system  for 
voicing  commands  to  non-data  link  aircraft. 


3 . 1 . 2 . 1 Data  Link  Identifiers  and  Pilot  Response  Indicator-  - 
The  matter  of  identifying  data  link  equipped  A/C  was  huuulid  by 
placing  an  asterisk  in  the  first  line  of  the  data  block  after  the 
ARTS  III  aircraft  identity.  Initially,  this  presented  a problem 
because  the  aircraft  identity  field  is  limited  to  seven  characters. 

. However,  it  was  easily  resolved  by  restricting  aircraft  identities 

to  five  characters  and  using  the  6th  character  position  for  the 
asterisk  designating  data  link.  The  remaining  7th  character  posi- 
tion was  thus  available  for  the  mutually  exclusive  data  link  indi- 
cations of  pilot  compliance  (W) , noncompliance  (U) , nonresponse 
(X),  and  link  failure  (F) . The  conditions  under  which  these  indi- 
cations were  displayed  were  as  follows: 


a.  A solid  W was  displayed  after  a programmed  time  delay 
following  controller  dispatch  of  a command  to  a data 
link  aircraft.  This  delay,  which  varied  + 3 seconds 
around  a mean  of  5,  was  automatically  generated  within 
the  ARTS  III  IOP  in  order  to  simulate  the  pilot's 
recognition  of  the  command  on  his  cockpit  display  and 
his  subsequent  depression  of  a WILCO  button.  Upon 
display  of  the  W,  the  command  that  had  been  WILCO' d 
was  erased.  The  W was  retained  on  the  display  for  3 
seconds  in  order  to  give  the  controller  a positive  in- 
dication of  the  aircraft's  compliance. 

b.  The  characters  U,  F,  or  X were  displayed  in  a blinking 
data  block  if  either  the  pilot  was  unable  to  comply  (U) 
with  the  controller-dispatched  command,  a failure  (F) 
occurred  in  the  data  link,  or  the  pilot  did  not  respond 
in  time  (X)  to  the  command.  In  such  cases,  both  the 
indicator  and  the  command  (nonblinking)  would  be 
retained  on  the  display  for  a period  of  time  (20  seconds) 
sufficient  for  the  controller  to  resolve  the  problem  on 
the  voice  link. 
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Disable 


Enable 


Halt 
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kestonnc  the 


of  MtiS  commands  for  an  air 


craft  against  which  a DISABLE  action  was  taken. 

Temporary  interruption  in  the  operation  of 
the  voice  response  system 


f.  Resume  - Restoration  to  operation  of  the  voice  response 
system  after  being  temporarily  halted. 

It  should  be  noted  that  the  Disapprove,  Disable,  and  Enable 
keyboard  functions  could  not  be  used  in  this  experiment  because 
of  the  inability  of  the  M5S  algorithm  to  dynamically  reschedule 
and  resequence  aircraft  or  to  update  commands  for  aircraft. 

3. 1.2. 3 Computer-Generated  Voice  - The  other  major  necessary  mod  if  i - 
cation  was  interfacing  the  ARTS  III  M§S  system's  computer-generated 
commands  with  a Voice  Response  System  (VRSj  (Reference  2)  which 
would  enunciate  commands  that  would  otherwise  be  voiced  by  the 
controller.  The  aim  here  was  to  permit  the  controller  to  use 
the  same  function  key  for  dispatching  both  voice  link  and  data 
link  commands,  thereby  providing  some  commonality  in  operating 
procedures  in  a mixed  voice/digital  communications  environment. 

Use  of  the  voice  response  system  would  also  provide  the  added 
advantages  of  speech  standardization  and,  more  importantly,  off 
loading  of  the  oral  communications  workload  of  the  controller  to 
the  extent  that  he  might  have  an  opportunity  to  consider  other 
commands  while  the  voice  response  system  was  speaking  for  him. 
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The  original  airspace  structure  employed  in  the  development 
of  the  ARTS  III  Mf, S system  (Reference  3)  was  based  on  that  existing 
at  Denver  Stapleton  International  Airport.  As  shown  in  Figure  3, 
aircraft  enter  the  terminal  area  through  three  pairs  of  feeder 
fixes.  Each  pair  of  feeder  fixes  funnel  the  aircraft  into  an 
associated  inner  fix  from  which  they  are  directed  onto  a final 
approach  path,  through  the  approach  gate,  and  on  to  the  runway. 
Traffic  flow  in  the  entire  approach  area  is  regulated  by  the  M$S 
algorithm  on  the  basis  of  flight  plan  data,  radar  position,  and 
aircrai't  performance  data.  Regulation  is  achieved  by  automatic 
sequencing  of  aircraft  through  the  feeder  fixes  and  thereafter 
by  patl  stretching  and  speed  control. 

Depending  on  type,  aircraft  are  scheduled  through  either  a 
high  performance  or  low  performance  gate  via  associated  downwind 
legs.  Since  only  a limited  amount  of  regulation  can  be  achieved 
by  path  stretching  or  speed  control,  almost  all  arrivals  must  be 
held  at  the  feeder  fix  to  derandomize  high  density  flow. 

The  algorithm  calculates  a feeder  fix  departure  time  for  each 
aircraft  with  the  expectation  that  the  aircraft  will  depart  within+1 
minute  of  that  time.  The  departure  is  carried  out  under  direct 
control  of  the  enroute  controller  but  is  seldom  on  schedule  because 
of  the  algorithm's  propensity  to  change  the  departure  time  at  any 
moment . 

Careful  evaluation  of  the  airspace  and  the  operation  of  the 
algorithm  within  the  airspace  resulted  in  the  identification  of 
several  problems  that  would  have  a deleterious  effect  on  the  data 
link  application.  These  problems  and  the  approaches  to  circumvent- 
ing them  are  as  follows: 

a.  The  northern  portion  of  the  M§S  geometry  had  critical 
vectoring  areas  in  which  the  algorithm  had  difficulty 
spacing  and  sequencing  aircraft.  Therefore,  only  that 
portion  of  the  airspace  involving  south  arrivals  onto 
runway  26L  through  the  Shawnee  and  Elizabeth  feeder  fixes 
was  used  in  this  experiment. 
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Figure  3.  Denver  Metering  and  Spacing  Geometry  (Runway  26L) 
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The  MfjS  program  issued  constant  time  updates  in  the  hold- 
ing patterns,  thus  making  it  difficult  to  depart  an  air- 
craft from  the  feeder  fix  within  + 1 minute  of  a constantly 
changing  departure  time.  Furthermore,  since  the  fixes 
are  outside  the  M§S  geometry,  commands  for  data  link  air- 
craft in  holding  patterns  could  not  have  been  dispatched 
or  voiced.  Holds  were  therefore  eliminated  by  careful 
structuring  of  traffic  samples. 


By  virtue  of  (a)  and  (b)  above,  the  role  of  the  enroute 
controller  in  the  scheduling  of  feeder  fix  departures, 
as  called  for  in  the  basic  ARTS  III  Mf)S , was  eliminated. 


The  resequencing  portion  of  the  M§S  program  was  not  able 
to  resequence  missed  approaches  back  into  the  flow  of 
traffic.  Subject  controllers  were  therefore  instructed 
to  keep  aircraft  in  sequence;  otherwise,  the  algorithm 
would  malfunction,  causing  conflicts,  large  spacing  gaps, 
and  holds.  In  addition,  provision  was  made  for  automatic 
acquisition  of  the  ILS  localizer  by  simulated  aircraft 
after  the  final  heading  command  had  allowed  the  aircraft 
to  intercept  the  localizer. 


If  M§S  commands  were  not  followed  by  controllers,  holding 
would  be  created  and  aircraft  would  get  out  of  sequence. 
Controllers  were  therefore  instructed  to  issue  all  com- 
mands, not  to  maneuver  aircraft  to  avoid  conflicts,  and 
not  to  use  the  Disapprove  and  Disable/Enable  keyboard 
functions.  They  were  also  instructed  to  notify  observers 
of  situations  in  which  they  (the  controllers)  felt  they 
would  normally  have  intervened.  These  situations  were 
recorded  for  future  analysis. 


In  the  basic  ARTS  III  M$S,  the  M§S  path  - stretching  geo- 
metry was  not  readily  discernible  to  the  controller. 
Therefore,  in  this  experiment,  a video  map  overlay  was 
included  on  the  controller's  display  to  provide  cues  on 


the  geometry. 


3.3  CONTROL  MODE  SELECTION 

The  candidate  control  modes  evaluated  in  this  program  were 
selected  on  the  basis  of  the  findings  from  a previous  controller/ 
computer  interface  experiment  conducted  at  the  NAFEC  Digital 
Simulation  Facility  (DSF).  In  the  DSF  experiment,  a simplified 
version  of  the  ARTS  III/M§S  system  was  simulated  in  order  to  eva- 
luate seven  candidate  modes  by  which  air  traffic  controllers  could 
dispatch  automatically  generated  commands  to  voice  link  and  data 
link  equipped  aircraft  in  terminal  approach  traffic.  The  seven 
modes  tested  at  the  DSF  were: 

a.  Control -by-Approval , Full  Data  Block,  Multiple  Command, 
Voice . 

b.  Control-by-Approval , Full  Data  Block,  Multiple  Command, 
Computer-Generated  Voice. 

c.  Control-by-Approval,  Tabular  List,  Multiple  Command, 

Voice 

d.  Control-by-Approval,  Tabular  List,  Multiple  Command, 
Computer-Generated  Voice. 


e.  Control -by-Approval , Tabular  List,  Single  Command,  Voice 

f.  Control-by-Approval,  Tabular  List,  Single  Command,  Com- 
puter-Generated Voice. 

g.  Control -by-Except ion , Full  Data  Block,  Multiple  Command, 
Computer-Generated  Voice. 

The  above  modes  represented  various  combinations  of  the  following 
design  alternatives: 


1.  Display  of  commands  in  the  full  data  block  or  the  tabular 
list. 

2.  Presentation  of  active  commands  one  at  a time  (single) 
or  in  multiple  fashion. 

3 Commun icat ion  to  voice  link  aircraft  by  computer-generated 
voice  or  by  controller  voice. 
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Controller  approval  and  dispatch  of  each  command  or 
automatic  dispatch  without  prior  controller  approval. 


. 


4. 

The  results  of  the  DSF  experiment  showed  that  the  most  pro 
mising  candidate  modes  of  operation  for  follow-on  testing  in  the 
operational  version  of  the  real  ARTS  III  M£S  system  at  the  NAH.C 
TATF  were  as  follows: 

a.  Control -by-Except ion , Full  Data  Block,  Multiple  Command, 
Computer  Generated  Voice. 

b.  Control-by-Approval , Tabular  List,  Single  Command, 
Computer-Generated  Voice. 

c.  Control-by-Approval,  Full  Data  Block,  Multiple  Command, 
Computer-Generated  Voice. 

d.  Control-by-Approval,  Full  Data  Block,  Single  Command, 
Computer-Generated  Voice. 

Although  not  tested  at  the  DSF,  the  fourth  mode  was  recom- 
mended because  it  represented  all  of  the  alternatives  that  the 


test  data  indicated  as  being  most  desirable,  particularly  the 
dispatching  of  commands  one  at  a time  without  the  requirement  for 
time-consuming  trackball  action.  However,  only  the  first  three 
modes  were  implemented  and  evaluated  in  the  real  ARTS  III  M£S 
system  at  the  TATF.  The  fourth  was  not  included  because  software 
modifications  would  have  been  required  to  provide  a single  command, 
full  data  block  capability  in  the  M$S  program,  and  it  was  felt 
that  the  design  of  the  existing  system  should  not  be  altered  to 
accommodate  a new  candidate  mode. 


4.  SYSTEM  DESCRIPTION 


4.1  GENERAL 

In  the  real  world,  an  ARTS  III  M§S  System  modified  for  opera- 
tion with  a data  link  channel  would  generate  and  transfer  heading, 
altitude,  and  speed  commands  to  a data  link  ground  system  (DLGS) . 
The  DLGS  would  uplink  the  commands  to  the  specified  data  link 
equipped  aircraft  and  would  accept  downlinked  responses  from  the 
pilot.  These  downlink  responses  would  then  be  transferred  to  the 
ARTS  III  M§S  system  for  display  to  the  controller. 

In  the  modified  ARTS  III  M§S  system  implemented  in  the 
Terminal  Automation  Test  Facility  at  NAFEC  (Figure  4)  all  functions 
other  than  the  controller/computer  interface  are  simulated.  All 
target  information  is  provided  by  the  ARTS  internal  target  gene- 
rator. An  ARTS  display  with  two  data  entry  devices  is  used  by 
"pilots"  to  make  necessary  target  maneuver  entries  for  non-data 
link  equipped  aircraft  in  response  to  commands  enunciated  by  either 
the  controller  or  by  a computer-generated  voice  system.  Responses 
by  simulated  data  link  equipped  aircraft  to  controller-dispatched 
commands  are  automatically  generated,  require  no  pilot  keyboard 
actions,  and  are  presented  to  the  controller  in  the  appropriate 
data  blocks  on  the  ARTS  III  Data  Entry  and  Display  System  (DEDS). 

4.2  OPERATIONAL  DESCRIPTION 

4.2.1  Keyboard  Functions 

The  keyboard  entries  incorporated  in  the  ARTS  III  DEDS  for 
use  by  the  controller  in  data  link  operations  are: 

4. 2. 1.1  Dispatch  - The  dispatch  entry  is  used  in  the  Control-by- 
Approval/Ful 1 Data  Block  (CBA/FDB)  and  Control -by-Approval/Tabular 
List  (CBA/TAB)  modes  to  indicate  approval  of  an  M5S  generated 
heading,  altitude,  or  speed  command.  In  the  CBA/FDB  mode,  the 
Dispatch  entry  entails  slewing  the  trackball  to  the  selected  target 
symbol  on  the  display  and  depressing  DEDS  function  16  key.  When 
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Figure  4.  Controller/Computer  Terminal  TATF  Simulating 
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this  entry  is  made,  all  blinking  commands  in  the  third  line  of  the 
specified  aircraft's  data  block  are  regarded  by  the  system  as 
approved.  If  the  entry  is  legal,  the  blinking  command (s)  are 
changed  to  a nonblinking  format  and  the  command(s)  are  transferred 
to  the  pilot's  display. 

In  the  CBA/TAB  mode,  the  Dispatch  entry  requires  only  depres- 
sion of  the  F16  key,  and  the  Dispatch  applies  only  to  the  command 
occupying  the  topmost  blinking  position  in  the  tabular  list. 

The  entry  is  acceptable  for  both  data  link  and  non-data  link 
equipped  aircraft.  If  the  entry  is  legal,  the  topmost  blinking 
command  is  changed  to  a nonblinking  format,  and  the  command  is 
transferred  to  the  pilot's  display. 


4. 2. 1.2  Disapprove  - The  Disapprove  entry  was  incorporated  for 
use  in  only  the  Control-by-Exception/Full  Data  Block  mode  to  over- 
ride an  M§S  generated  heading,  altitude,  or  speed  command.  However, 
its  use  in  the  experiment  was  disallowed  because  of  shortcomings 

in  the  M§S  algorithm.  The  Disapprove  action  involves  the  same 
keyboard  entry  as  for  Dispatch  (i.e.,  trackball  slew  and  depression 
of  the  F16  key)  and  applies  to  all  blinking  commands  in  the  third 
line  of  the  specified  aircraft's  full  data  block.  The  Disapprove 
entry  results  in  erasure  of  the  disapproved  blinking  command(s). 

If  no  blinking  commands  are  displayed  at  the  time  the  action  is 
taken,  the  next  command(s)  for  the  selected  aircraft  is  disapproved. 
In  the  latter  case,  the  command(s)  is  displayed  nonblinking  and 
is  thus  ineligible  for  dispatch.  Disapproved  commands  are  not 
transferred  to  the  pilot's  display. 

4. 2. 1.3  Disable/Enable  - The  Disable  entry  was  originally  incor- 
porated for  use  in  any  of  the  control  modes  to  inhibit  temporarily 
data  link  use  of  M$S  generated  commands  pertaining  to  a specified 
aircraft.  However,  as  in  the  case  of  Disapprove,  its  use  in  the 
experiment  was  disallowed.  The  entry  entails  depression  of  the 
F12  key  followed  by  a trackball  slew  and  enter  action.  A subse- 
quent F12  slew  entry  identifying  the  same  aircraft  constitutes  an 
Enable  action  which  allows  the  aircraft  to  again  receive  data  linked 
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commands.  Commands  for  a "Disabled"  aircraft  are  displayed  non- 
blinking, are  ineligible  for  Dispatch,  and  are  not  transferred 
to  pilot's  display. 


4. 2. 1.4  VRS  Control  - The  VRS  Control  entry  is  used  to  temporarily 
interrupt  the  VRS,  permitting  controller  use  of  the  voice  channel. 
The  entry  requires  despression  of  the  F13  key  for  either  disabling 
or  reenabling  the  VRS,  depending  on  its  current  state  as  indicated 
in  the  controller's  preview  area  by  either  "VRS  ON"  or  "VRS  OFF." 

4.2.1. 5 Handof f- to -Tower  Command  - In  all  modes  employing  the  VRS, 

a Handof f-to-Tower  command  can  be  issued  to  both  data  link  and  voice 
link  aircraft  by  means  of  a sequence  of  keyboard  entries:  F12,  H 

(for  Handoff),  Trackball  Slew,  and  Enter  action  by  the  controller. 
For  voice  link  aircraft  only,  these  entries  will  activate  the  VRS 
which  will  enunciate  the  following  message  preceded  by  the  aircraft 
ID  for  the  aircraft  in  question:  "Now  X miles  from  outer  marker, 

cleared  for  ILS  runway  two  six  left,  approach.  Contact  Denver 
tower  on  one  one  eight  point  three  when  over  outer  marker." 


For  voice  link  and  data  link  aircraft,  the  aircraft's  position 
symbol  is  changed  to  a "T,"  the  FDB  blinks  for  several  seconds, 
after  which  the  aircraft  is  under  control  of  position  "T." 

4.2. 1.6  Control  Mode  Selection  - The  Mode  Selection  entry  is  used 
to  designate  the  desired  mode  of  operation:  CBA  or  CBE;  Voice  or 

VRS.  The  entry  requires  an  F12,  two  characters,  and  Enter.  Accept- 
able character  pairs  include  the  following: 

AS  for  CBA/VRS 
ES  for  CBE/VRS 
AV  for  CBA/Voice 

The  method  of  command  presentation,  full  data  block  or  tabular 
list,  is  selected  on  the  control  panel  of  the  ARTS  III  IOP. 


4.2.2  Control  Modes 


The  sequence  of  events  associated  with  an  M$S-generated  com- 


r1 


- < 


mand  is  shown  in  Figure  5.  The  command  is  generated  with  the 
expectation  that  it  will  be  executed  by  the  aircraft  at  some  future 
time.  Shortly  after  it  is  generated,  the  command  is  displayed 
blinking  to  the  controller  in  either  the  third  line  of  the  data 
block  or  in  a tabular  list,  as  shown  in  Figure  0.  In  the  latter 
case,  the  aircraft's  FDB  position  symbol  is  also  blinked  when  the 
command  becomes  the  topmost  blinking  command  in  the  list.  In 
either  command  presentation  mode,  the  blinking  commands  are  eligible 
for  controller  Dispatch  or  Disapprove  actions  and  will  continue  to 
blink  until  such  action  has  been  taken  or  issue  time  has  been 
attained.  When  issue  time  has  been  reached  in  control -by-approval 
modes,  the  command  cannot  be  dispatched;  however,  in  control-by- 
exception, the  command  is  automatically  dispatched.  At  issue  time 
or  at  the  time  of  Dispatch  (CBA  Mode),  the  command  stops  blinking 
and  is  retained  until  termination  or  until  a pilot  (WILCO)  response 
is  received.  There  are  different  command  presentation  parameter 
values  associated  with  each  of  the  three  control  modes  tested  in 
this  experiment.  The  three  modes  are  discussed  in  detail  below 
in  terns  of  their  respective  parameter  values. 


i 


4.2.2. i Control -by-Approval/Ful  1 Data  Block/Voice  Response  System 
(CBA/FPB/VRS)  - In  the  CBA/FDB/VRS  mode,  the  M5S  command  is  dis- 
played blinking  in  the  third  line  of  the  aircraft's  FDB  12  seconds 
before  the  issue  time.  Figure  7 shows  the  controller's  display  in 
the  CBA/FDB/VRS  mode  of  operation  with  an  active  altitude  command 
for  voice  link  aircraft  AF14  and  a W(WILCO)  response  for  data  link 
aircraft  TT73.  The  controller  dispatches  the  command  by  slewing 
the  trackball  to  the  appropriate  controller  symbol  and  depressing 
the  F16  function  key.  The  dispatched  command  stops  blinking  and 
is  displayed  with  the  associated  ACID  in  a list.  Two  or  three 
blinking  commands  for  the  same  aircraft  can  be  transferred  simul- 
taneously with  a single  dispatch  action.  Since,  from  this  point 
on,  there  are  different  event  sequences  for  data  link  and  non-data 
link  aircraft,  resepct ivel y , each  is  separately  described  below. 

a.  Data  Link  Aircraft  - If  the  dispatched  command  is  for  a 
data  link  aircraft,  a suitably  delayed  response  (W,U,F,  or  X)  is 
automatically  generated  ami  displayed  in  the  ACID  line  of  the 
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aPpropriate  FDB  on  the  controller's  display  and  adjacent  to  the 
appropriate  ACID  and  command  on  the  pilot's  display.  The  W(Wilco) 
and  U (Unable)  delays  vary  +3  seconds  about  a 5-second  mean,  with 
a standard  deviation  of  1 second,  and  are  intended  to  reflect 
pilot  reaction  time,  equipment  delays,  sampling  delay,  and  other 
types  of  message  delays.  (These  values  are  based  on  measurements 
reported  in  Reference  4.)  An  F (link  failure)  indication  was  set 
at  one  second,  an  X (timeout)  at  14  seconds.  Display  of  a W(Wilco) 
results  in  erasure  of  the  command,  indicating  automatic  execution 
of  the  command  by  the  internal  target  generator.  The  W is  erased 
3 seconds  after  it  is  initially  displayed.  Display  of  a U,  F,  or 
X response  results  in  the  first  two  lines  of  the  data  block  blink- 
ing for  14  seconds  to  alert  the  controller  that  he  must  intervene 
on  the  voice  link  to  issue  orally  the  vectoring  command  that  has 
not  been  or  could  not  be  executed  by  the  aircraft.  By  displaying 
the  command  and  computer-generated  responses  for  data  link  aircraft 
on  the  pilot's  display,  the  pilot  is  able  to  monitor  the  situation 
and  thus  be  prepared  to  communicate  orally  with  the  controller 
when  the  need  arises. 

b.  Voice  Link  Aircraft  - If  the  dispatched  command  is  for  a 
voice  link  aircraft,  it  is  enunciated  by  the  VRS  and  orally  acknow- 
ledged by  the  pilot,  who  also  makes  a keyboard  entry  to  allow  the 
target  generator  to  execute  the  command.  The  command  is  retained 
in  the  FDB  on  the  controller's  display  for  20  seocnds  after  the 
Dispatch  action.  It  is  erased  from  the  pilot's  display  at  the 
time  the  pilot  makes  the  keyboard  entry. 

4. 2. 2. 2 Control -by-Approval/Tabular  List/Voice  Response  System 
(CBA/TAB/VRS)  - In  this  mode,  all  new  commands  are  presented  blink- 
ing at  the  bottom  of  the  tabular  list  12  seconds  before  issue  time. 
Figure  8 shows  the  controller's  display  in  the  CBA/TAB/VRS  mode  of 
operation.  When  a command  is  the  topmost  blinking  command  in  the  list, 
it  is  eligible  for  dispatch,  and  the  associated  aircraft's  position 
symbol  starts  blinking  to  provide  the  controller  with  a visual  cue 
for  command-aircraft  correlation.  The  controller  dispatches  the 
command  by  merely  depressing  the  F16  function  key  — a trackball 
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slew  action  is  not  required.  When  the  Dispatch  action  is  taken, 
the  command  and  position  symbol  stop  blinking,  and  the  command  is 
transferred  to  the  pilot's  display. 

The  event  sequence  following  the  Dispatch  action  is  the  same 
as  in  the  CBA/FDB/VRS  mode;  i.e.,  the  command  will  be  erased  from 
the  controller's  list  after  20  seconds  for  a voice  link  aircraft 
or  upon  display  of  pilot's  WILCC(W)  response  in  the  ACID  line  of 
the  FDB  for  a data  link  aircraft. 

4. 2.2.3  Control-by-Exception/Full  Data  Block/Voice  Response  System 
(CBE/FDB/VRS)  - In  this  mode,  commands  are  displayed  blinking  in 
the  third  line  of  the  FDB  and  are  automatically  dispatched  6 seconds 
later,  unless  the  controller  makes  a Disapprove  entry  (trackball 
slew,  F16)  during  the  blinking  interval.  In  the  latter  case,  the 
command  is  immediately  erased.  If  the  controller  made  a Disapprove 
entry  with  no  blinking  commands  present,  the  entry  will  apply  to 
the  next  command,  which  will  appear  nonblinking  for  5 seconds 
before  being  erased.  As  previously  mentioned  in  Section  4. 2. 1.2, 
use  of  the  Disapprove  entry  was  disallowed  in  the  experiment. 

4.2.3  Pilot  Keyboard  and  Display 

As  previously  mentioned,  a separate  Data  Entry  and  Display 
Subsystem  (DEDS)  with  two  data  entry  devices  was  incorporated  in 
the  modified  ARTS  system  to  accommodate  two  operators  functioning 
as  simulation  pilots.  For  each  pilot,  the  display  presented  a 
separate  list  (see  Figure  9)  that  contains  the  ACID  and  a single 
character  identifier.  When  commands  are  dispatched  by  the  con- 
troller, they  appear  in  the  list  on  the  pilot  display  in  association 
with  the  ACID  and  identifier.  For  data  link  aircraft,  the  response 
indicator  is  displayed  beside  the  ACID.  If  the  response  is  a W, 
the  command  is  erased  and  the  W retained  for  3 seconds.  If  the  res- 
ponse is  a U,  F,  or  X,  the  command  is  retained  and  the  pilot  can 
expect  to  engage  in  voice  communication  with  the  controller  to 
resolve  the  problem.  For  voice  link  aircraft,  the  VRS  enunciates 
the  command  to  the  pilot,  and  the  pilot  must  (1)  voice  his  com- 
pliance, and  (2)  enter  the  single  character  identifier  on  his  key- 
board. Upon  entry  of  the  identifier,  the  target  generator  executes 
the  command,  and  the  command  is  erased  from  the  pilot's  display. 
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5,  TEST  CONDUCT 


5.1  CONTROLLER  PROCEDURES 

In  both  test  series  conducted,  each  of  three  controllers 
tested  each  of  the  three  data  link  control  modes  against  three 
traffic  mixes;  i.e.,  201,  501,  and  80%  data  link  aircraft.  Each 
controller  also  tested  the  two  control-by-approval  modes  without 
the  simulated  data  link  and  VRS;  i.e.,  voice-only  operation.  Prior 
to  the  start  of  the  test  series,  a detailed  set  of  procedures  was 
distributed  to  each  of  the  subject  controllers.  The  procedures 
explained  the  traffic  situation  and  contained  step-by-step  in- 
structions of  the  operation  in  each  of  the  three  modes.  The 
subject  controllers  were  also  orally  briefed  on  all  aspects  of  the 
experiment  and  then  allowed  to  observe  the  operation  of  the  display 
during  a 1-hour  general  training  session.  In  addition,  each 
subject  received  a 1-hour  training  run  immediately  before  a set 
of  evaluation  runs  with  the  mode  of  operation  to  be  tested.  The 
schedule  for  each  subject's  assignment  to  these  runs  was  kept  in 
strict  accordance  with  the  test  matrix  described  in  Section  6. 

5.2  CONTROLLER  OBSERVER  PROCEDURES 

During  each  training  and  evaluation  run,  the  subject  was 
observed  by  another  controller  familiar  with  the  simulation  design 
and  operation.  The  function  of  the  observer  was  to  verify  that  an 
acceptable  level  of  operation  was  achieved  with  respect  to  all 
aspects  of  an  evaluation  run.  The  observer  made  entries  in  a run 
log  whenever  any  peculiarities  were  observed,  equipment  mal- 
functions occurred,  or  actual  conflicts  were  noted  by  the 
controller,  and  when  the  controller  failed  to  dispatch  commands. 

The  observer  determined  whether  or  not  a test  should  be  rerun  if 
in  his  judgment  a breakdown  had  occurred  in  on-line  data  collection 
because  of  the  subject  controller's  failure  to  observe  required 
procedures.  The  decision  to  rerun  a test  run  was  confirmed  through 
review  of  an  end-of-run  summary  printout  shown  in  Figure  IP. 
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5.3  PILOT  PROCEDURES 

Two  simulator  pilots  were  employed  in  each  test  run  to  "fly" 
the  aircraft  from  the  horizontal  ARTS  DEDS.  As  previously 
mentioned,  the  pilot  keyboard  entries  and  verbal  acknowledgements 
were  limited  to  non-data  link  aircraft.  Commands  to  data  link 
aircraft  were  displayed  on  the  pilot  display  but  were  automatically 
processed  (acknowledged)  by  the  simulation  software  without 
pilot  intervention,  thus  eliminating  uncontrolled  pilot  errors 
during  a test  run.  The  data  link  communications  environment  for 
the  controller  was  simulated  by  incorporating  the  effects  of  the 
following  two  types  of  delays  between  dispatch  and  receipt  of  an 
associated  response:  (1)  a real  pilot's  reaction  time  to  a com- 

mand on  a cockpit  I/O  device,  and  (2)  uplink  and  downlink  trans- 
mission time  of  commands.  The  reaction  time  of  a real  pilot  was 
simulated  by  a normal  distribution  of  time  delays  that  were 
activated  at  random  by  each  command  such  that  W delays  would 
vary  +3  seconds  about  a 5-second  mean  with  a standard  deviation 
of  1 second.  Although  the  transmission  delays  were  generally 
smaller  than  the  W delays,  they  were  included  for  realism  because 
they  represent  the  following: 

a.  Equipment  delays  in  ground  and  airborne  transceivers 

b.  Delays  due  to  uplinking  of  other  messages  (ATIS,  weather 
advisories,  etc.)  when  a high  priority  command  was 
received  by  the  data  link  ground  systems. 

c.  Sampling  delays  during  the  repeated  ground  system 
interrogation  of  an  aircraft  for  its  W response. 

For  each  non-data  link  aircraft  command  displayed  and  enunciated 
by  the  VRS,  the  simulator  pilots  were  required  to  (1)  enter  a 
single  alphanumeric  ACID  character  which  would  allow  the  simulated 
aircraft  to  execute  the  command,  and  (2)  verbally  acknowledge  the 
command  with  a "Roger."  The  single  character  ACID  identifier  was 
quickly  entered  by  the  pilot  so  that  the  controller  would  hear  a 
rapid  pilot  acknowledge.  The  rapidity  of  the  pilot  responses 
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were  considered  important  to  a fair  comparison  of  voice  and  data 
link  message  transaction  times  inasmuch  as  real  pilots  tend  to 
abbreviate  or  hurry  their  responses  when  the  voice  channel  is 
busy . 


5.4  TEST  DATA 

The  subjective  and  objective  test  data  from  this  experiment 
were  derived  from  controller  questionnaires  and  test  data  re- 
trieved on  line  by  the  ARTS  III  computer  during  each  test  run, 
respectively.  The  questionnaires  were  categorized  as  follows: 

a.  A pre-experiment  questionnaire  completed  after  briefing 
and  before  any  training  runs  to  ascertain  the  subject's 
attitudes  toward  data  link  and  the  experiment. 

b.  A post-training  questionnaire  answered  after  the 
completion  of  training  on  a particular  control  mode, 
but  before  any  test  runs  for  that  mode.  This  question- 
naire sought  to  obtain  controller  opinion  about  training 
adequacy  as  well  as  pre-test  opinion  about  the  performance 
of  the  mode  to  be  tested. 

c.  A post-test  questionnaire  filled  out  after  the  completion 
of  each  test  run  to  obtain  controller  opinion  about  the 
mode  just  tested,  including  overall  performance  during 
the  test  run,  and  the  operational  characteristics  of  the 
mode  tested. 

d.  A post-series  questionnaire  given  after  completion  of  all 
test  runs  for  a particular  mode  to  determine  if  there  were 
differences  in  controller  reaction  with  respect  to  the 
three  traffic  mixes. 

e.  A summary  questionnaire  answered  after  each  controller 
had  completed  all  the  test  runs  for  all  the  modes,  in- 
tended to  provide  data  on  controller  preference  and  rating 
of  the  modes  with  respect  to  such  aspects  as  stress, 
blunders,  and  workload. 
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The  test  data  collected  by  the  ARTS  III  were  reduced  and 
processed  by  data  extraction  programs  after  the  test  runs  were 
completed.  The  extraction  programs  measured  the  following  types 
of  data: 

1.  Time  intervals  between  successive  communications  actions 

2.  Counts  of  significant  events 

3.  Aircraft  positions  for  each  scan 

The  measurements  required  for  the  first  two  types  of  data 
were  defined  by  means  of  constructing  event  time  lines  (see  the 
appendix)  for  every  sequence  of  communications  actions  that 
occurred  for  each  of  the  three  modes.  The  key  time  intervals 
associated  with  the  two-way  transaction  of  each  message  are  de- 
picted in  the  time  lines.  The  time  interval  values  obtained  were 
used  in  the  comparative  analysis  of  the  three  modes.  The  key 
intervals  were: 

a.  Message  Transaction  Time  (MTT)  : The  total  time  betw'een 

display  of  an  M§S  command  to  the  controller  to  the 
completion  of  a normal  pilot  response. 

b.  Command  Initiation  Delay  (CID) : The  time  between  command 

display  and  initiation  of  controller  action.  Initiation 
was  defined  as  occuring  when  the  command  was  dispatched. 

c.  Service  Time  (ST) : The  time  required  to  service  an  active 

command  that  had  become  the  topmost  blinking  command  in 
the  tabular  list. 

d.  Queue  Delay  (OP) : The  time  required  for  a command  to 

reach  the  topmost  position  in  the  tabular  list  after 
being  initially  displayed. 

The  time  lines  illustrate  the  importance  of  the  command  in- 
itiation delay  as  a component  of  total  message  transaction  time 
and  as  a measure  of  controller  workload.  Since  one  important  way 
in  which  the  CID  was  thought  to  manifest  itself  was  in  the  format 
of  a queue  of  commands,  an  expanded  list  of  MTT  and  command  queue 
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measurements  were  included  in  a data  reduction  program  summary 
printout  (see  Figure  11) . The  most  important  of  these  were  the 
total  times  for  all  voice  and  data  link  message  transactions  and 
the  average  queue  size. 

Another  set  of  objective  data  obtained  during  each  test  run 
consisted  of  counts  of  significant  events,  such  as  normal  voice 
link  and  data  link  command,  exception  cases,  extemporaneous  com- 
munications, handoffs,  etc.  The  total  number  of  these  events  was 
also  listed  in  each  summary  printout  as  an  aid  in  determining  the 
validity  of  a test  run.  A total  count  of  the  number  of  manual 
tag  repositionings  was  also  obtained  to  gain  a measure  of  controller 
workload  and  distraction  from  his  normal  communications  task. 
(Automatic  tag  offset  was  disabled  by  controller  preference.)  In 
addition  to  the  above  data,  the  track  position  each  radar  scan 
of  all  aircraft  was  extracted  for  subsequent  input  to  conflict 
detection  and  analysis  programs  that  were  exercised  after  the  test 
runs  were  completed. 


The  observer  controllers  recorded  the  aircraft  identifications 
and  the  time  of  occurrence  whenever  they  were  alerted  by  the  sub- 
ject controllers  that  a conflict  situation  was  developing  in  which 
the  aircraft  would  normally  be  maneuvered.  Similarly,  the  pilots 
recorded  any  instances  in  which  the  subjects  transmitted  commands 
not  generated  by  the  M§S  algorithm  and  when  commands  timed  out 
without  controller  dispatch. 


5 . S TRAFFIC  SAMPLES 


The  prime  objective  in  the  design  of  the  traffic  samples  was 
to  subject  the  controller  to  a linear  increase  in  traffic  density 
from  zero  to  nine  aircraft  over  a test  period  of  1 hour. 

Objective  data  were  collected  for  a 45  minute  period  from  this 
hour.  The  traffic  density  in  this  case  was  defined  as  the 
controller's  instantaneous  aircraft  count  in  the  region  between 
the  two  feeder  fixes  and  the  point  of  handoff  to  the  tower.  The 
period  of  1 hour  was  deemed  sufficient  to  provide  results  re- 
presentative of  an  operational  situation.  The  increase  in  aircraft 
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load  to  nine  during  the  hour  was  intended  to  facilitate  the 
continuous  measurement  of  the  controller’s  communication  delays 
which,  it  was  theorized,  would  slowly  increase  until  his  workload 
limit  was  reached  and  then  increase  rapidly  thereafter. 

To  develop  a traffic  sample  which  provided  a linearly  in- 
creasing traffic  load  as  a function  of  time,  an  initial  sample 
consisting  of  30  aircraft  was  prepared  which  approximated  the 
desired  pattern.  The  initial  sample  had  to  he  changed  experi- 
mentally until  the  desired  results  were  obtained.  This  could  not 
be  done  analytically  because  the  Mf,S  algorithm  adapted  to  changes 
in  the  traffic  sample  by  changing  times  at  which  commands  were 
issued  or  by  issuing  speed  commands.  Operational  parameters  also 
had  to  be  changed.  For  example,  the  minimum  allowable  separation 
had  to  be  reduced  from  3 to  2 miles  to  accommodate  the  maximum  of 
nine  aircraft  for  the  controller's  instantaneous  load.  Also,  to 
circumvent  operational  deficiencies  of  the  Mf)S  algorithm,  the 
aircrafl  types  were  limited  to  high  performance  jets  f"07)  only; 
jumbo  jets  and  low  performance  aircraft  were  specifically  excluded 

The  procedure  used  in  developing  a master  traffic  sample  con- 
sisted o •'  conducting  numerous  planning  runs  during  which  the  entry 
and  exit  times  of  each  aircraft  were  recorded.  From  that  informa- 
tion a chart  of  the  aircraft  load  as  a function  of  time  was  con- 
structed. This  chart  was  examined  and  changes  in  the  aircraft 
entry  tires  were  made  to  improve  the  linearity  of  the  increasing 
aircraft  load.  This  process  was  repeated  many  times  until  a point 
was  reached  where  the  recommended  changes  causes  adaptations  by 
the  MfiS  algorithm  which  adversely  affected  linearity.  The  traffic 
sample  selected  was  the  one  which  immediately  preceded  the 
occurence  of  adverse  effects  (charted  in  Figure  12).  The  portion 
of  the  sample  from  10  to  55  minutes  was  used  for  data  collection 
purposes . 

Since  it  was  impractical  to  create  11  different  traffic 
samples  by  this  process,  the  same  basic  sample  was  employed  in 
all  experiment  runs  by  each  controller.  To  minimize  learning 
effects,  the  aircraft  identifications  and  the  occurrence  of  data 
link  failure  events  were  changed  from  run  to  run. 
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It  addition,  a separate  training  sample  was  constructed. 

This  sample  had  a slightly  lower  average  traffic  load,  and  while 
it  generally  increased  as  the  run  progressed,  no  attempt  was  made 
to  make  that  sample  increase  linearly  with  time. 


6.  TFST  DESIGN 


This  section  discusses  the  determination  of  the  test  settings 
and  the;  sequence  of  runs  scheduled  for  the  simulation  study. 

The  objective  of  the  study  was  to  determine  differences  between  the 
three  candidate  modes  as  a function  of  the  percentage  or  mix  level  of 
data  link  vs.  voice  link  equipped  aircraft.  A secondary  con- 
sideration to  provide  voice-only  data  against  which  the  other 
testing  could  be  compared  was  added  late  in  the  planning  stages  of 
the  experiment. 

The  development  of  the  experimental  plan  was  influenced  by 
several  factors: 

a.  Rather  than  limit  the  experiment  to  one  level  of  air 
traffic  or  increase  the  number  of  runs  by  introducing 
load  as  an  additional  factor,  it  was  decided  to  use  a 
dynamic  density  variation  that  increased  linearly  with 
the  test  time. 

b.  Each  test  mode  was  to  be  handled  by  as  many  subject 
controllers  as  possible  to  prevent  the  relative  evalua- 
tion of  modes  from  being  distorted  by  differences  ir.  the 
skills  of  individual  controllers. 

c.  The  order  in  which  the  modes  and  percentages  were  tested 
by  the  subject  controllers  would  contaminate  the  results 
unless  the  plan  was  set  up  to  balance  ovit  such  effects. 

For  example,  if  each  controller  tested  three  modes,  it 
must  be  assumed  that  his  performance  on  the  third  test 
would  be  better  than  his  performance  on  the  first  be- 
cause of  his  experience  on  tests  one  and  two.  To 
compensate  for  this,  the  sequence  of  tests  was  assigned 
so  that  each  mode  and  percentage  was  tested  by  one 
controller  for  whom  it  was  the  first,  by  one  controller 
for  whom  it  was  his  second,  etc.  In  statistical 
terminology  this  technique  is  called  BLOCKING  in  the 
sense  that  the  run  sequence  is  treated  as  a block  effect. 


d.  A basic  unit  of  three  controllers  testing  all  the 

nine  combinations  of  the  three  modes  and  three  percentage 
levels  would  permit  blocking  of  the  mode  and  percentage 
sequences.  This  basic  unit  could  be  repeated  with  a dif- 
ferent set  of  controllers  to  obtain  greater  statistical 
power  in  the  analysis. 

e.  While  the  traffic  situation  simulated  represented  a 
fairly  standard  ATC  situation,  the  manipulation  of  the 
function  keys  and  slewball  varied  from  mode  to  mode. 

To  insure  that  each  controller  was  operating  at  an 
acceptable  level  of  proficiency  and  to  familiarize  him 
with  the  specific  procedures  for  the  mode  being  tested, 
sufficient  training  had  to  be  provided.  To  minimize  the 
training  time  all  the  percentage  runs  for  one  mode  were 
to  be  grouped  and  preceded  by  a one  hour  training  run. 

f.  To  handle  the  addition  of  the  voice-only  runs  in  a 
structured  fashion  would  greatly  enlarge  the  required 
number  of  runs  for  a basic  unit.  If  the  analysis  were 
limited  to  simple  comparisons  between  voice-only  and 
percentage  run  averages,  then  appropriate  runs  could  be 
inserted  at  random  to  the  groups  of  runs  for  each  mode. 

The  above  considerations  dictated  the  use  of  a factorial  de- 
sign for  the  test  variables  (i.e.,  mode,  percentage,  and  subject) 
with  the  percentage  and  mode  sequences  blocked  in  a Greco-Latin 
square.  Using  this  arrangement,  the  test  runs  were  sequenced  and 
the  training  and  voice-only  runs  inserted,  as  shown  in  Table  1. 


RUN  SCHEDULE 


7.  SUBJECTIVE  DATA  ANALYSIS 


Analysis  of  the  subjective  (questionnaire)  data  for  the  Data 
Link  Phase  2B  experiment  consisted  of  a tabulation  of  the  responses 
to  each  question  by  mode  and  percentage  mix.  These  tabulations 
were  analyzed  to  determine  if  there  were  any  meaningful  differences 
between  the  modes  or  the  mixes.  In  addition,  where  similar 
questions  were  given  at  different  points  in  the  experiment,  those 
data  were  analyzed  to  determine  if  the  responses  differed. 

The  results  of  this  analysis  are  reported  by  question.  Each 
question  is  given  exactly  as  stated  in  the  questionnaire.  Following 
each  question  is  a discussion  of  the  findings.  The  responses  to 
each  question  were  subjectively  reviewed  to  determine  if  there  was 
any  consistent,  sizable  (i.e.,  meaningful)  differences  attributable 
either  to  the  mode  or  percentage  effects.  Only  meaningful  dif- 
ferences are  reported.  If  the  mode  or  mix  differences  are  not 
meaningful,  no  comment  is  made,  and  generally  the  results  are 
aggregated  across  the  nonmeaningful  effect (s).  All  important 
findings  in  this  section  are  underlined.  A statistical  analysis  of 
these  data  was  not  performed  due  to  the  limited  sample  size  oi  six 
sub j ects  . 

The  questionnaires  consisted  of  five  parts:  (1)  the  Pre- 

Experiment  questionnaire  given  after  briefing  and  before  any 
operational  exposure,  (2)  the  Post-Training  questionnaire  given 
after  completion  of  the  one  training  hour  for  each  run,  (3)  the 
Post-Test  questionnaire  given  after  the  completion  of  each  test 
hour,  (4)  the  Post-Series  questionnaire  given  after  completion  of 
all  runs  of  a mode,  and  (5)  the  Summary  questionnaire  given  after 
completion  of  all  runs. 

7.1  PRE-EXPERIMENT  QUESTIONNAIRE 

The  Pre-Experiment  questionnaire  was  given  after  oral  briefing 
and  before  any  operational  exposure.  The  questionnaire  consisted 
of  seven  questions  intended  to  ascertain  the  subjects  attitudes 
toward  data  link  and  the  experiment  as  a whole. 


1.  Do  you  understand  the  objectives  of  the  data  l' ■it 
experiments? 

Yes  100%  No  0% 

Ai i of  the  subjects  felt  that  they  understood  the  objec t i v e of 
the  data  link  display  experiment. 

2.  Choose  a statement  describing  the  degree  of  mdt 

the  data  link  study  which  you  achieved  by  re  iding  t l tumenta- 

tior  provided . 

a.  Complete  and  comprehensive  50% 

b.  Workable  with  minor  questions  50% 

c.  Marginal  with  major  questions  0% 

d.  Vague  and  confused  0% 

All  of  the  subjects  felt  they  had  a workable  understanding 
of  the  study  after  reading  the  documentation  provided. 

3.  Do  you  feel  that  the  semi-automation  of  terminal  ATC  computer 
generated  metering  and  spacing  commands  transmitted  via  aata 
link  will  eventually  be  accepted  in  actual  operations ? 

a.  Yes  831 

b.  do  0% 

c.  Undeci ded  17% 

five  of  the  six  subject  controllers  indicated  that  they  felt 
the  semi -automat  ion  of  terminal  ATC  computer  generated  metering  and 
spacing  commands  transmitted  via  data  link  would  be  accepted  in 
actual  operation. 

4.  How  do  you  anticipate  that  the  addition  of  computer  generates 
metering  and  spacing  commands  transmitted  via  data  link  will 
affect  the  current  operational  system? 


a . 

(+2) 

Very  favorably  0% 

b. 

( + 1) 

Favorably  66% 

a. 

( 0) 

Hot  very  much  17% 

d. 

(-1) 

Unfavorably  17% 

e . 

'-2) 

Very  unfavorably  0% 
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Most  of  the  subjects  felt  that  the  addition  of  computer 
generated  metering  and  spacing  commands  transmitted  via  data  link 
would  favorably  affect  the  current  operating  system.  One  controller 
felt  the  addition  of  a data  link  would  not  have  very  much  effect  on 
the  system  while  another  one  felt  the  effect  would  be  unfavorable. 
Using  the  scores  indicated  in  parentheses  next  to  the  responses 
gives  an  average  response  of  0.5,  indicating  a mildly  favorable 
attitude . 


Rank  those  aspects  of  the  upcoming  test  uhich  g u feel  Z.  ast 
confident  about  (uhere  the  aspect  about  uhich  y u -ire  least 
confident  is  numbered  1): 

a.  Performance  at  maximum  traffic  loads 

b.  Mix  of  data  link  and  voice  linn  airaraf 
a.  Metering  and  spacing  algorithm 

d.  Display  formats 

e.  I/O  procedures 

f.  Preventing  the  tag  overlaps 

g.  Maintaining  safe  separation 

h.  Trackball  operations 

i.  VRS  procedures 


The  responses  to  this  question  are  tabulated  in  Table  2.  A 
shown  by  the  rank  sum  column,  the  metering  and  spacing  algorithm 


was  clearly  the  most  troublesome  aspect  of  the  test  prior  to 


experimentation.  Performance  at  maximum  traffic  load  was  the 


second  most  troublesome  aspect,  having  received  first  place 
ranking  by  four  of  the  subjects  and  near  last  place  by  the  other 
two.  The  mix  of  voice  and  data  link  aircraft,  trackball  operation 
and  VRS  procedures  were  the  least  troublesome. 


Did  you  participate  in  the  Data  Link  Phase  PA  experimt  < t 
the  DDF? 


Five  of  the  six  subjects  had  participated  in  the  Phase 
experiment,  indicating  that  a comparison  between  the  Phase 
the  Phase  2B  results  for  similar  questions  is  appropriate. 


TABLF.  2.  TABULATION  OF  RESPONSFS 
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7.  Plt’.ase  indicate  how  recently  (in  years)  you  have  had  experience 
with  the  following  systems  (if  no  experience  indicate  with  a 
0)  . 

a.  Prototype  M&S  on  the  TATF 

b.  Operational  ARTS 

c.  Other  TATF  experiments 

Only  one  subject  had  experience  with  the  prototype  Mf,S  on 
the  TATF,  and  that  experience  was  within  one  year.  Only  one 
other  controller  had  operational  ARTS  exposure,  and  that  was 
two  years  ago.  One  other  subject  had  worked  on  other  TATF 
experiments  two  years  ago.  While  only  half  of  the  subjects  had 
direct  ARTS  experience,  all  the  subjects  had  considerable 
experience  on  other  DSF  experiments  which  utilize  a digital, 
ARTS-like  display. 

7.2  POST-TRAINING  QUESTIONNAIRE 

The  Post-Training  questionnaire  was  given  after  the  second 
training  run  and  prior  to  any  test  exposure  for  that  mode.  The 
objective  of  the  questionnaire  was  to  ascertain  the  controllers' 
opinions  about  the  adequacy  of  the  training  and  their  prerun 
opinions  about  the  performance  of  the  mode  being  tested. 

1.  Were  the  principal  features  of  the  mode  demonstrated  clearly 
in  the  training  runs?  Yes  1001  No  01 

All  the  subjects  felt  that  the  modes  were  clearly  demonstrated 

2.  How  much  more  training  do  you  think  you  would  need  to  feel 
comfortable  with  the  procedures? 

a . One  hour  1 00 » 

b.  Three  hours  01 
a.  Five  hours  01 

d.  Ten  hours  0% 

All  of  the  responses  indicated  that  a minimum  amount  of 
additional  training  (1  hour)  would  have  been  desirable. 


— 


How  do  you  anticipate  that  this  mode  of  displaying  and 
transmitting  commands  will  effect  the  current  operational 
sys tern? 

a.  ( + 2)  Very  favorably 

b.  (+ 1 ) Favorably 

c.  ( 0)  Not  very  much 

d.  (- 1 ) Unfavorably 

e.  (-2)  Very  unfavorably 

The  responses  to  this  question  are  tabulated  by  mode  below: 


RESPONSE 


Very  favorably  (+2) 
Favorably  (+1) 

Not  very  much  (0) 
Unfavorably  (-1) 

Very  unfavorably  (-2) 


Average  score 


As  can  be  seen  above,  the  subject  controllers  had  a favorable 
opinion  of  the  CBE  mode,  a mildly  unfavorable  opinion  of  the 
TAB  mode,  and  a strongly  unfavorable  opinion  of  the  FOB  mode 
rior  to  the  test  runs.  - 

. How  do  you  anticipate  that  this  mode  of  displaying  and  trans- 
mitting commands  will  effect  the  following  characteristics  of 
the  current  operating  system? 


ASPECT 


Workl oad 

Traffic  Handling 
Capac i ty 

Succept  ibi 1 i ty 
to  Blunders 

Stressfullness 


GREATLY 

DECREASED  DECREASE 


WILL  NOT 
CHANGE 


INCREASE 


GREATLY 

INCREASE 


Using  the  scores  indicated  in  parentheses,  the  responses  to 
this  question  have  been  reduced  and  are  shown  below: 


ASPECT 


Workload 

Traffic  Handling  Capacity 
Susceptibility  to  Blunders 
Stressfulness 
Total 


These  scores  indicate  an  opinion  very  similar  to  the  overall 
grade  given  in  Question  3.  The  CBE  mode  is  given  a mildly 
favor  able  response,  TAB  mildly  unfavorable  and  FDB  strongly 
un  favorable . 

7.3  POST-TESTING  QUESTIONNAIRE 

The  Post-Testing  Questionnaire  was  given  after  the  completion 
of  each  run.  It  was  composed  of  three  series  of  questions , A,  B, 
and  C . 

7.3.1  A-Series 

The  questions  of  the  A-Series  were  designed  to  solicite 
general  controller  attitude  toward  the  mode  tested. 

A - 1 . A.  How  do  you  feel  that  this  node  of  displaying 

transmitting  commands  would  affect  the  current 
operational  system  for  the  same  traffic  situation? 

a.  (+ 2 ) Very  favorably 

b.  (+1)  Favorably 

c.  ( 0)  Not  very  much 

d.  (-1)  Unfavorably 

e.  (-2)  Very  unfavorably 

The  responses  to  this  question  were  reduced  using  the  scores 
shown  in  parentheses  next  to  the  responses.  These  results  are 
shown  on  the  next  page. 


MODE 


PERCENTAGE  DL 


SUM  OF  EXCLUSIVE  OF 
01 


:a 


0 

20 

50 

80 

FDB 

1 

-5 

-5 

-5 

-15 

TAB 

-3 

0 

-3 

-7 

-10 

CBE 

XX 

3 

2 

2 

7 

TOTAL 

-2 

-6 

-10 

These  results  summed  across  percentages  indicate  the  same 
relationship  between  the  modes  as  did  Post -Training  Questions 
3 and  4 . CBE  would  have  a mildly  favorable  impact  on  the 
operational  system;  TAB  unfavorable;  and  FDB  more  strongly 
unfavorabl e when  summed  across  the  modes.  The  results  indicate 
a trend  which  shows  an  increasingly  unfavorable  attitude  as  the 
percentage  of  data  link  equipped  aircraft  is  increased. 

A-l  . B.  For  the  above , what  ratings  would,  you  give  each  of 


the  following  aspects? 


GREATLY 

WILL  NOT 

GREATLY 

ASPECT 

DECREASE 

DECREASE 

CHANGE 

INCREASE 

INCREASE 

Workload 

f + 2) 

( + 1) 

(0) 

f - 1 j 

(-2) 

Traffic  Handling 
Capacity 

(-2) 

(-D 

(0) 

(+1) 

(+  2) 

Susceptibility 
to  Blunders 

(+  2) 

( + 1) 

(0) 

(-1) 

(-2) 

Stressfulness 

(+  2) 

( + D 

(0) 

(-1) 

(-21 

Using  the  scores  indicated  in  parentheses,  the  subject  re- 
sponses were  reduced  to  give  the  results  shown  in  Table  3.  The 
mode  totals  indicate  the  same  relationship  shown  previously  in 
Post-Training  Questions  3 and  4 and  Post-Test  Question  A-l. A.  The 
subjects  felt  that  CBE  would  have  a generally  favorable  impact  on 
current  operations  while  TAB  would  have  an  unfavorable  impact  and 
FDB  an  even  more  unfavorable  impact . 


The  favorable  rating  for  CBE  came  from  an  expected  improvement 
in  the  workload  and  traffic  handling  capacity  aspects,  while  the 
susceptibility  to  blunders  and  stressfulness  aspects  as  expected 
was  unchanged.  The  unfavorable  reaction  to  TAB  primarily  reflected 
an  unfavorable  response  to  the  susceptibility  to  blunders  and  the 
stressfulness  characteristics.  The  unfavorable  response  to  FOB  was 
contributed  to  by  unfavorable  reactions  in  all  aspects. 

The  trend  of  increasingly  unfavorable  responses  with  in- 
creasing percentage  observed  in  Post-Test  Question  A-l.A  was  not 
as  evident,  which  indicates  that  the  controllers'  negative  reaction 
to  increasing  DL  employment  could  not  be  explained  in  terms  of  the 
individual  control  aspects.  The  voice-only  runs  were  graded  some- 
what  more  favorably  than  the  data  link/VRS  combination  runs. 


Once  again,  the  responses  to  this  question  indicate  the  same 
relationship  between  modes  as  shown  in  Post-Training  Questions 
3 and  4 and  Post-Test  questions  A-l.A  and  A-l.B.  CBF.  in  an  all 
data  link  operation  was  judged  to  affect  the  current  operation 
favorably;  TAB  was  judged  unfavorably;  and  FDB  even  more  un- 
favorably . The  magnitude  of  the  responses  to  this  question  was 
very  similar  to  those  for  Question  A-l.A,  indicating  very  little 
expected  difference  due  to  an  all  data  link  environment. 

A. 2 B.  For  the  above , what  ratings  would  you  give  the  following 
aspects? 


ASPECT 

Workload 

Traffic  Handling 
Capac ity 

Susceptibility 
to  Blunders 

Stressfulness 


GREATLY 

DECREASE 

DECREASE 

WILL  NOT 
CHANGE 

INCREASE 

GREATLY 

INCREASE 

(+2) 

( + 1) 

CO) 

(-1) 

(-2) 

(-2) 

(-D 

(0) 

( + D 

( + 2) 

( + 2) 

( + D 

(0) 

(-1) 

(-2) 

(+  2) 

( + 1) 

(0) 

C-l) 

(-2) 

The  responses  to  Question  A- 2 . B are  given  in  Table  4 reduced 
by  the  scores  indicated  in  the  parentheses  above.  The  responses 
to  this  question  indicate  the  same  relationships  shown  previously 
in  Post-Training  Questions  3 and  4 and  Post-Test  Questions  A-l.A, 
A-l.B,  and  A-2.A.  In  an  all  data  link  environment,  CBF  was  con- 
sidered to  affect  the  current  operation  favorably;  TAB  was  judged 
unfavorably;  and  FDB  even  more  unfavorably . 

When  compared  to  Question  A-l.B,  the  responses  are  of  similar 
magnitude  indicating  that  no  disadvantage  would  accrue  to  an  all 
data  link  environment.  In  addition,  the  contribution  by  aspect  to 
the  favorable  or  unfavorable  reaction  on  each  mode  was  about  the 
same.  For  CBF,  the  favorable  response  was  recorded  in  the  work- 
load and  traffic  handling  capacity  aspects , while  the  susceptibility 
to  blunders  and  stressfulness  aspects  were  about  neutral.  For 
FDB  and  TAB,  the  unfavorable  response  was  recorded  in  all  four 
aspects . 
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7.3.2  B -Series 

The  B-Series  contained  questions  which  applied  to  all  of  the 
modes.  The  first  three  B questions  were  concerned  with  the 
controller's  attitude  toward  his  preparation  for  and  performance 
during  the  run. 

B.l  Now  that  you  have  been  tested  in  this  mode , how  much  more 

training  time  do  you  think  you  would  need  to  feel  comfortable 
with  the  procedures? 

a.  One  hour 

b.  Three  hours 

c.  Five  hours 

d.  Ten  hours 

The  results  of  this  question  have  been  summarized  by  mode  and 
shown  below: 


TIME 

MODE 

FDB 

TAB 

CBF. 

One  hour 

11 

16 

12 

Three  hours 

1 

Five  hours 

Ten  hours 

8 

4 

Average 

4.7 

2.8 

1.0 

Only  five  of  the  six  subjects  elected  to  answer  this  question. 
In  several  other  cases,  a response  was  written  in  which  indicated 
an  opinion  outside  the  range  of  the  answers  such  as  "none"  or 
"never."  In  these  cases,  the  most  appropriate  box  either  one  hour 
or  ten  hours  was  assigned  the  response. 

The  responses  show  much  the  same  opinion  as  indicated  in 
Post-Training  Questions  3 and  4 and  in  the  A-series  questions. 

The  controllers  felt  very  comfortable  with  the  CBF.  mode  and  would 
have  required  only  one  additional  hour  of  training.  In  several 
instances,  in  the  TAB  mode,  the  controllers  felt  very  uncomfortable 
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and  the  average  rose  to  2.8  hours.  For  FDB,  there  were  even  more 
cases  where  the  controllers  felt  uncomfortable  and  the  average 
rose  even  higher  to  4.7  hours. 


B-2.  Rank  those  aspects  of  the  test  run  which  gave  you  the  most 

trouble  (where  the  aspect  giving  the  most  trouble  is  numbered 

1): 

a.  Performance  at  maximum  traffic  load 

b.  Mix  of  data  link  and  voice  link  aircraft 

c.  Metering  and  spacing  algorithm 

d.  Display  formats 

e.  I/O  procedures 

f.  Preventing  tab  overlaps 

g.  Maintaining  safe  separations 

h.  Trackball  operations 

i.  VRS  procedures 

In  response  to  this  question,  several  controllers  ranked  only 
those  aspects  which  were  troublesome.  To  normalize  the  responses, 
ranked  aspects  were  assigned  the  average  unused  ranks;  e.g.,  if 
aspect  a was  ranked  1,  b ranked  2,  and  c ranked  3,  then  aspects  d 
through  i would  have  been  assigned  the  average  of  ranks  4 through 
9 or  6.5.  In  cases  where  aspects  were  given  the  same  rank,  they 
were  arbitrarily  assigned  that  rank  plus  a factor  equal  to 
(n-l)/2,  where  n is  the  number  of  co-ranked  aspects.  For  example, 
if  aspects  d,  e,  and  f had  been  assigned  rank  2,  then  that  rank 
would  have  been  changed  to  2+(3-l)/2  = 3.  If  necessary,  lower 
ranks  were  adjusted  to  correct  for  ranks  occupied  by  ties.  Fol- 
lowing this  process  the  ranks  were  accumulated  to  give  the  re- 
sults shown  in  Table  5.  The  scores  for  the  voice  runs  are 
multiplied  by  3 since  3 times  as  many  VRS  runs  were  made. 

As  can  be  seen  from  that  table,  for  all  modes,  the  MSS 
algorithm  was  a major  problem.  For  the  FDB  mode,  preventing  tag 
overlaps  and  trackball  operations  were  very  significant.  For  the 
TAB  mode,  no  additional  factors  were  important.  For  CBF,  the 
display  format  and  the  prevention  of  tag  overlaps  were  somewhat 


TABLE  5.  ACCUMULATED  RANKS  FROM  POST-TEST  OUESTION 


A low  score  is  indicative  of  a more  troublesome  aspect. 
The  number  in  the  upper  righthand  corner  gives  the  rank 
aspect  for  that  mode. 
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Tlie  rankings  and  grades  for  the  Voice  and  VRS  runs  were  very 
similar  except  in  the  FDB  mode  where  the  prevention  of  tag  overlaps 
become  very  much  more  bothersome  in  the  voice  modes  due  to  the 
requirement  to  read  the  third  line  in  the  data  block  to  transmit 
the  command. 

In  comparison  with  Question  5 of  the  Pre-Experiment  question- 
naire, the  responses  are  very  similar  except  that  the  concern  over 
performance  at  maximum  traffic  load  decreases  considerably  after 
the  run.  This  result  duplicates  the  findings  of  the  Phase  2A 
experiment . 

B-3.  Which  of  the  following  statements  bes t describes  your  feelings  at 

vhe  end  of  the  test  run  just  concluded? 

a.  The  aircraft  load  was  relatively  easy  to  handle 

b.  You  could  control  a small  number  of  additional 

aircraft  safely 

c.  You  were  operating  at  your  maximum  capacity 

d.  You  were  somewhat  overloaded 

e.  You  were  badly  overloaded 

The  results  of  this  question  are  tabulated  by  mode  in  the 
following  table: 


RESPONSE 

HUB 

TAB 

CBE 

a . 

Easy  to  handle 

O 

o 

o\° 

20.9% 

33.4% 

b. 

Near  capacity 

8.4% 

12.5% 

22.2% 

c . 

At  capacity 

45.81 

2 5.0% 

22.2% 

d. 

Somewhat  overloaded 

16.7% 

29.1% 

11.1% 

The  remaining  B questions  were  designed  to  ascertain  the 
controllers'  opinions  of  engineering  or  operating  characteristics 
of  the  experiment. 

B-4.  Were  any  of  the  characters,  symbols,  or  actions  employed 
in  this  mode  particularly  difficult  to  interpret? 

Yes  471  No  531 

Almost  half  of  the  subjects  had  difficulty  with  certain 
characteristics  of  the  modes.  This  difficulty  was  not  dependent 
upon  the  mode  but  upon  the  individual.  The  controllers  did  not 
specify  which  of  the  characters,  symbols,  or  actions  were  dif- 
ficult to  interpret. 

B-5.  Were  you  able  to  evaluate  each  command  carefully  before  you 
dispatched  it? 

a.  Almost  always  14% 

b.  In  a majority  of  cases  20% 

c.  About  half  the  time  30% 

d.  In  a minority  of  cases  19% 

e.  Rarely  171 

There  was  considerable  variation  in  response  to  this  question 
An  examination  of  the  mode  percentage  of  differences  did  not  show 
any  pattern.  However,  in  the  majority  of  instances,  the  control- 
lers did  not  feel  that  they  were  able  to  evaluate  all  commands 
prior  to  dispatch. 

B-6.  Were  you  able  to  maintain  the  "picture"  of  the  overall  traf- 
fic situation  and  monitor  it  for  potential  conflicts  even 
while  transmitting  commands  at  the  maximum  rate? 

a.  Almost  always  14% 

b.  In  a majority  of  cases  13% 
a.  About  half  the  time  17% 

d.  In  a minority  of  cases  13% 

e.  Rarely  34% 

As  in  Question  B-5,  there  was  a varied,  unpatterned  response. 
However,  in  this  case  the  responses  clustered  around  the  extreme 
points . 


B-7.  How  do  you  rate  your  confidence  in  being  able  to  detect 
and  resolve  any  conflicts  that  might  occur  while  using 
this  mode? 

a.  Extremely  high 

b.  High 

a.  Moderate 

d.  Low 

e.  Very  pessimistic 

As  shown  below,  the  responses  to  this  question  show  the 
familiar  modal  pattern: 


RESPONSE 


Extremely  high 
High 

Moderate 

Low 

Very  pessimistic 


With  the  CBE  mode,  the  controllers'  confidence  in  being  able 
to  detect  and  resolve  conflicts  is  high  to  moderate:  with  the  TAR 


mode  moderate  to  low;  and  with  the  FDR  mode,  low. 


B-8.  Was  the  MSS  conflict  list  useful  m monitoring  the  traffi 
situation? 

Yes  17%  No  831 

Only  17%  of  the  controllers  felt  the  conflict  list  was 
useful  . This  was  no  doubt  due  to  the  unreliability  of  the  Mf,S 
conflict  prediction  algorithm. 

B-9.  Did  the  MSS  conflict  list  distract  you  from  other  more 
important  tasks? 

Yes  17%  No  83% 

Only  one  of  the  six  controllers  felt  that  the  MfiS  conflict 


list  was  distract in 


MODE 

FDB 

TAB 

CBE 

13% 

13% 

33% 

13% 

21% 

6% 

4% 

16% 

22% 

37% 

13% 

28% 

33% 

37% 

11% 

use  the  automatic 


The  controllers  used  the  automatic  tag  repositioning  feature  in 


runs 


Most  of  the  subjects  preferred  to  use  the  size  2 setting 
(.136  inch/character);  however,  during  maximum  traffic  load,  many 
controllers  used  setting  1 (.084  inch/character).  The  responses 
are  not  tabulated  since  many  of  the  subjects  checked  more  than  one 
response. 


The  C-Series  questions  pertain 
individual  modes  and  were  given  to 
completion  of  a run  of  that  mode. 


Questions  C-l  to  C-5  pertain  only  to  the  CBE  and  FPB  modes, 
where  the  command  information  is  contained  in  the  full  data  block 


did.  the  command  line  in  the.  full  data  bio 
clutter  to  the  point  that  your  ability  to 
was  impaired? 

Yes  62%  No  38% 


Most  of  the  subjects  felt  that  the  increased  display  clutter 
impaired  their  ability  to  control  traffic.  Prelocation  of  tags  by 
approach  phase  was  not  employed  as  in  the  Phase  2A  experiment. 


did  the  clutter  cause  the  command  line  to  be  obscured  to  the 
point  where  it  caused  operational  difficulties  or  an  exces- 
sive number  of  tag  repositioning  ? 

Yes  83%  No  17% 
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reposi t ionings  . 


C-3.  Tf  you  used  the  automatic  offset  feature,  did  it  cause  you 
to  mi  • s any  of  the  commands? 

Yes  67%  No  33% 

This  question  was  answered  only  three  times.  In  two  of  these 
cases  the  controllers  felt  that  the  automatic  offset  feature  did 
cause  them  to  miss  commands. 

C-4.  Was  the  presentation  and  coding  of  the  commands  in  the  full 
data  block  easily  interpreted? 

Yes  76%  No  24% 

Approximately  1/4  of  the  subjects  had  difficulty  interpreting 
the  presentation  and  coding  of  the  commands  in  the  full  data  block 
Controllers  did  not  state  the  reasons  whv  they  had  difficulty; 
however,  that  difficulty  was  probably  caused  by  the  somewhat 
cluttered  display. 

C-5.  Did  you  have  any  difficulty  in  recognizing  an  unable  (U ) , a 
link- fail  (F),  or  a no  response  (X)  indication  to  act  on  it 
in  a sufficiently  timely  fashion? 

Yes  76%  No  24% 

Almost  80%  of  the  subjects  had  difficulty  in  recognizing  the 
exception  condition  indications  to  act  on  them  in  a sufficiently 
timely  fashion.  Once  again,  this  difficulty  was  probably  caused 
by  the  cluttered  display. 

Question  C-6  applies  only  to  the  CRA/FDB  command  mode. 

C-6.  When  the  number  of  commands  waiting  to  be  transmitted  builds 
up,  do  you  feel  that  it  would  be  acceptable  to  dispatch 
several  commands  in  rapid  succession  without  evaluating 
each  command? 

Yes  22%  No  78% 
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Question  C-10  was  dropped  after  the  questionnaires  were 
printed.  The  question  was  ambiguously  worded  and  the  subjects 
were  instructed  to  ignore  it. 

Questions  C-ll  through  C-14  apply  to  the  CBE  mode  only. 

C-ll . Were  provisions  for  halting  one!  starting  tht 

for  maintaining  discipline  on  the  voice  channel? 

Yes  501  No  50% 


E ? 


for  halting  and  starting  the  VRS  were  needed.  No  detailed  informa 
tion  was  provided  by  the  subjects. 

C-12.  Did  you  find  it  disconcerting  not  to  have  pre-transmission 
approval  of  issued  commands? 

Yes  441  No  561 


Almost  half  of  the  responses  indicated  that  not  having  pre- 
transmission approval  of  issued  commands  was  disconcerting . 


C-13.  Was  the  3-second  gap  between  VRS  transmissions 


a.  Too  long  29% 

b.  About  right  531 
a.  Too  short  181 

Most  of  the  controllers  felt  that  the  3-second  gap 
between  VRS  transmissions  was  appropriate. 


C-14.  Stx-seconds  were  allowed  to  disapprove  each  command.  Was 


this  time 


a.  Too  long  38% 

b.  About  right  54% 

c.  Too  short  8% 

Most  of  the  responses  indicated  that  the  6-second  dis- 
approval time  was  about  the  right  length.  However,  a fairly 
large  percentage  felt  it  was  somewhat  too  long. 


Question  C-15  applies  only  to  the  control -by-approval  modes. 


C-15.  If  a command  is  not  dispatched  within  12  seconds,  the  com- 
mand stops  blinking.  Twenty  seconds  after  that , the  common d 
is  erased.  Did  this  feature  cause  anu  problems  during  the 


The  responses  to  this  question  varied  by  mode  as  shown  in 
the  table  below. 


V ■/*.’ 


RESPONSE 


MODE 

FDB 

TAB 

78  % 
22% 

58  % 
4 21 

In  both  modes,  the  majority  of  responses  indicated  that  the 


time-out  of  commands  caused  problems.  However,  in  the  FDB  mode 


almost  twice  as  many  responses  as  in  the  TAB  mode  indicated  a 


problem . This  was  probably  caused  by  the  increased  time  required 
to  detect  a flashing  command  and  to  slew  to  the  aircraft  symbol  for 
dispatch . 


Questions  C-16  to  C-19  apply  to  VRS  runs  only. 

C-16.  did  you  find  any  of  the  following  aspects  of  the  VRS 
bothersome ? 

a.  Intelligibility  1 

b.  Voice  quality  2 
3.  Talking  rate  14 

d.  Inability  to  repeat  command  20 

■?.  Interference  with  your  own  extemporaneous  messages  9 

The  freauency  wi th  which  each  aspect  was  checked  is  indicated 
next  to  each  response.  As  can  be  seen,  the  pre  lominant  problems 
were  first,  the  inability  to  repeat  commands  and  secondly,  talkim 


rate.  Interference  with  extemporaneous  commands  was  a mild 
problem . 

C-17.  Dc  you  find  the  dispatching  an.;  monitoring  of  VRS  commands 
less  tiring  than  voicing  the  commands  yourself? 

Yes  6M  No  331 

Two-thirds  of  the  responses  indicated  that  the  VRS  was  less 
tiring  than  voicing  the  commands. 
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C-18.  Pid  you  listen  to  the  VRS  deliver  its  message  and  to  the 
pilot's  reply  before  shifting  your  attention  to  the  next 
command? 

a.  Almost  always  17% 

b.  In  a majority  of  oases  28% 
a.  About  half  the  time  23% 

d.  In  a minority  of  oases  13% 

e.  Rarely  19% 

The  responses  to  this  question  were  fairly  equally  divided 
among  the  choices.  No  pattern  was  discernible  in  either  the 
modes  or  percentages. 

C-19.  Commands  to  all  voice  link  aircraft  remain  displayed  for 
20  seconds  after  a successful  dispatch.  Should  that  time 
b e : 

a.  Lengthened  0% 

b.  Remain  the  same  43% 

c.  Shortened  57% 

Over  half  of  the  responses  indicated  that  the  commands  to 
voice  link  aircraft  should  remain  displayed  for  less  than  20 


seconds  after  a successful  dispatch, 


Questions  C-20  through  C-22  pertain  to  the  response  indica- 
tion for  data  link  commands  and  apply  only  to  the  VRS/data  link 
mix  runs. 

C-20.  The  WILCO  acknowledge  (W)  symbol  was  displayed  nonblinking 
for  3 seconds  after  the  receipt  of  the  WILCO.  do  you  feel 
that  time  should  be: 

a.  Lengthened  6% 

b.  Remain  the  same  41% 
a.  Shortened  53% 

Most  of  the  responses  indicated  that  the  3-second  nonblinking 
display  of  the  W symbol  after  receipt  of  a WILCO  could  be 
shortened.  The  6%  of  the  responses  for  lengthening  that  time  came 
from  the  CBE  mode;  however,  even  there  the  majority  were  for 
shortening  the  display  time. 
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C-21.  Was  the  display  of  the  WILCO  acknowledgement  (W)  symbol 

necessary  or  would  the  erasure  of  the  command  be  an  adequate 
indicator  of  successful  dispatch? 

a.  The  WILCO  symbol  was  required  11% 

b.  Erasure  would  be  adequate  89% 


Almost  90%  of  the  responses  indicated  that  erasure  of  the 
WILCO  acknowledgement  would  be  an  adequate  indicator  of  successful 
dispatch . 

C-22.  Considering  that  link  failures , unables , and  no  responses 
are  exception  cases  and , as  such , generally  are  more  time 
consuming , do  you  feel  that  an  excessive  amount  of  time  or 
effort  is  required  to  resolve  these  exception  cases? 

Yes  7%  No  93% 

Only  7%  of  the  responses  felt  that  exception  cases  required 
an  excessive  amount  of  time  or  effort  to  resolve. 

Questions  C-23  and  C-24  pertain  to  the  extrapolation  of  the 
operation  of  the  modes  under  different  failure  conditions. 

C-23.  Envision  the  following  situation:  Computer  aided  metering 

and  spacing  with  data  link  suddenly  becomes  inoperative 
during  the  peak  load  of  this  test . However,  you  still 
have  the  current  ARTS  III  display  and  procedures . In 
your  opinion: 

a.  Could  you  continue  operations  indefinitely  at  the  same 
traffic  level?  Yes  48%  No  52% 

b.  Could  you  land  all  the  aircraft  under  your  control 
safely  if  no  further  aircraft  were  accepted  in  the 
terminal  area?  Yes  97%  No  3% 

c.  If  no  further  aircraft  were  accepted  could  you  clear 
the  terminal  area  of  traffic  safely?  ves  95%  No  5% 

d.  A hazardous  and  unpredictable  situation  would  exist? 

Yes  9%  No  91% 
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About  half  of  the  responses  indicated  that  if  M§S  with  data 
link  failed,  they  could  continue  indefinitely  at  the  same  traffic 
level  if  the  current  ARTS  III  capabilities  were  available.  Almost 
all  of  the  responses  indicated  that  they  felt  they  could  clear 
the  terminal  area  or  land  all  aircraft  under  those  conditions. 

Less  than  101  felt  a hazardous  situation  would  exist. 

C-24.  Envision  the  following  situation:  Computer  aided  metering 

and  spaaing  with  data  link  and  ARTS  III  data  block  suddenly 
beaomes  inoperative  during  the  peak  load  of  this  test.  You 
would  be  left  with  only  the  primary  radar  targets . If  you 
had  the  current  capability  to  see  flight  strips  of  your 
targets  and  to  request  squawk  ID: 

a.  Could  you  continue  operations  indefinitely  at  the  same 
traffic  level?  Yes  7%  No  931 

b . Could  you  land  all  the  aircraft  under  your  control 
safely  if  no  further  aircraft  were  accepted  in  the 
terminal  area?  Yes  401  No  60% 

a.  If  no  further  aircraft  were  accepted  could  you  clear 
the  terminal  area  of  traffic  safely?  Yes  38%  No  62% 
d.  A hazardous  and  unpredictable  situation  would  exist? 

Yes  89%  No  11% 

As  would  be  expected,  when  left  with  only  primary  radar, 
squawk  ID,  and  flight  strips,  all  the  subjects  felt  the  situation 
would  be  much  worse.  Less  than  10%  felt  they  could  continue 
operations  indefinitely,  while  only  40%  felt  they  could  clear  the 


area  or  land  the  aircraft  even  if  no  further  aircraft  were  accepted. 
Almost  90%  felt  a hazardous  situation  would  exist. 


7.4  POST-SERIFS  QUESTIONNAIRE 

The  Post-Series  questionnaire  was  given  after  each  controller 
completed  all  the  runs  for  a mode.  The  questions  were  designed 
to  ascertain  differences  between  mix  percentages. 


1.  Did  you  feel  that  there  was  a significant  operational  dif- 
ference between  the  four  runs  just  completed?  Yes  611  No  .39% 

Slightly  over  601  of  the  responses  indicated  that  there  was 
a difference  between  runs.  No  comments  were  made  by  the  subjects 
on  the  nature  of  the  differences. 


If  yes t could  you  rank  those  runs  according  to  their  ease  of 
operation  (rank  the  easiest  run  1). 

a.  First  run 

b.  Second  run 
a.  Third  run 
d.  Fourth  run 


The  sum  of  the  ranks  by  mode  and  percentage  are  given  below: 


PERCENTAGE 


MODE 

FDB 

TAB 

CBE 

WITH 

WITHOUT 

WITH 

WITHOUT 

wItHQUT 

01 

0% 

0% 

0% 

01 

14.0 

12.5 

15.0 

11.5 

16.5 

12.5 

11.0 

16.0 

12.5 

15.5 

11  . 5 

13.0 

15.0 

12.0 

15.5 

12.0 

12.0 

TOTAL 
V ITHOU1 
01 


The  modes  were  ranked  in  a manner  similar  to  that  described 
in  Post-Test  Question  B-2.  As  can  be  seen  in  the  table,  the  0% 
runs  are  scored  somewhat  better  than  the  mixed  runs . This  is 
probably  due  to  the  greater  familiarity  that  controllers  have  with 
voice-only  procedures.  There  is  no  apparent  difference  between 
the  various  levels  of  mix. 


To  which  of  the  following  factors  do  you  attribute  the  observed 
differences? 

a.  Changes  in  the  traffic  sample  1 

b.  Previous  exposure  with  this  mode  2 

a.  Mix  of  data  link  and  voice  link  aircraft  3 

d.  Different  aircraft  loads  0 

e.  Use  of  the  VRS  1 


MODE 


Sum  of  Above  / 4 


The  number  of  times  each  factor  was  checked  is  listed  next  to 
the  choice.  The  question  was  asked  18  times;  as  can  be  seen, 
none  of  the  factors  seem  very  prevalent.  The  mix  of  data  link  and 
voice  link  aircraft  is  checked  most  frequently,  but  there  is  no 
apparent  difference  from  this  factor  as  seen  in  Question  2,  above. 


Using  the  sum  of  the  ranks  as  discussed  in  Post-Test 
Question  B-2,  scores  for  each  mode  were  developed  and  are  shown 
in  Table  6. 


TABLE  6.  SUMMARIZATION  OF  RESPONSES  TO  SUMMARY  OtJF.STION  1 


7.5  SUMMARY  QUESTIONNAIRE 

The  Summary  Questionnaire  was  given  after  each  subject  had 
completed  all  his  runs.  The  objective  of  this  questionnaire  was 
to  ascertain  the  controllers  comparison  of  the  modes. 

1.  Please  rank  (1  indicates  most  preferred)  the  three  modes 

which  you  operated  on  each  of  the  characteristics  listed  below. 


CHARACTERISTIC 


Workload 


Capacity 

Susceptibility  to  Blunders 
Stressfulness 


13-1/4 


Overall 


OVERALL 


WORKLOAD  CAPACITY 


CBA/TAB/VRS 

CBA/FDB/VRS 


CBE/FDB/VRS 


In  terms  of  the  individual  characteristics,  the  CBE  mode 
ranks  well  in  all  characteristics.  The  TAB  mode  ranks  poorly 
in  workload  and  capacity  and  well  in  susceptibility  to  blunders. 

The  FDB  mode  ranks  poorly  in  all  categories,  particularly  in  terms 
of  stressfulness . Combining  the  scores  of  the  individual  character 
istics  gives  the  same  order  to  the  modes  as  seen  in  most  previous 
responses,  CBE  best,  TAB  intermediate,  and  FDB  last. 

In  looking  at  the  overall  rating,  the  FDB  mode  and  the  CBE 
mode  are  ranked  equally,  and  the  TAB  mode  worst.  This  ranking  is 
contrary  to  all  other  indications.  One  possible  cause  for  this 
rating  is  that  when  the  controllers  were  asked  for  an  overall 
evaluation,  the  preference  for  an  FDB  display  overrode  their 
previous  operational  difficulties. 

2.  Ho w do  you  feel  that  the  availability  of  a data  link  in  each 
of  the  following  nodes  will  effect  controller  workload? 


STRONGLY  MILDLY  NO  MILDLY  STRONGLY 

INCREASE  INCREASE  EFFFCT  DECREASE  DECREASE 


CBA/TAB/VRS 


CBA/FDB/VRS 


CDF/FDB/VRS 


Using  the  scores  indicated  in  parentheses  above,  the  modes 
achieve  the  following  scores:  FDB=-8,  TAB=-10,  and  CBE=-7. 

Once  again,  as  in  the  overall  evaluation,  the  FDB  mode  is  not 
rated  as  poorly  as  in  the  other  questionnaires. 

3.  How  would  you  rate  realism  of  the  scenario  and  simulation? 

a.  Acceptable  as  is  501 

b.  Adequate  with  minor  problems  17% 

c.  Poor  with  major  problems  17% 

d.  Totally  inadequate  17% 

Half  of  the  subject  controllers  felt  that  the  realism  of  the 


scenario  and  simulation  was  acceptable.  The  other  half  each 


checked  one  of  the  other  responses. 


8.  ADDITIONAL  CONTROLLER  COMMENTS 

In  addition  to  checking  the  responses  on  the  questionnaire, 
the  controllers  often  expressed  their  attitude  toward  certain 
aspects  of  the  test.  Many  of  these  comments  explained  the  parti- 
cular rationale  behind  a judgment,  while  others  emphasized  situa- 
tions where  the  subjects  felt  very  strongly  about  an  issue. 

Several  of  the  comments  that  typify  the  general  reaction  or 
clearly  point  out  problems  or  recommended  improvements  have  been 
selected  and  are  discussed  in  the  following  paragraphs.  In  some 
cases,  the  comments  have  been  paraphrased  to  define  the  issue 
being  discussed. 

In  the  general  evaluation  of  the  modes,  the  key  troublesome 
issues  were  clearly  pointed  out.  For  the  Full  Data  Block  (FDB) 
mode  among  the  comments  were  "Trackball  functions  are  too  time 
consuming  — completely  unsafe  for  peak  traffic  conditions.'."'  and 
"Too  busy  slewing  and  button  pushing,  occupies  both  hands  — un- 
acceptable for  ATC  — unsafe."  For  the  Tabular  List  (TAB)  mode, 
the  comments  included:  "TAB  List  is  no  good  — constantly  requires 

double  checking  back  and  forth  from  TAB  List  to  aircraft"  and  "Too 
busy  looking  at  and  keeping  up  with  TAB  List  — very  poor."  For 
the  Control -by- Except  ion  (CBF.)  mode,  the  controllers  found  it 
disconcerting  not  to  have  pretransmission  approval  of  commands, 
"especially  when  commands  were  issued  which  caused  a conflict." 

For  the  FDB  voice-only  runs,  one  subject  stated  that  he  "was 
frequently  unsure  that  commands  had  been  transmitted,"  indicating 
a need  for  a manual  action  to  change  the  status  of  voice  dispatched 
commands  in  some  manner 

Very  frequently,  the  controllers  commented  on  being  paced  by 
the  computer.  Comments  such  as:  "Attention  basically  to  machine 

insti actions "At  peak  of  test,  1 felt  compelled  to  push  buttons 
without  proper  command  elevation  and  this  bothers  me."'  "Confusion 
reigns  with  four,  five,  or  six  simultaneous  commands,"  and, 
"Controller  is  racing  to  keep  up  with  the  computer,"  indicate 
strong  pressure  not  to  let  the  queue  of  unattended  messages  get 


too  long.  These  comments  were  most  often  expressed  for  the  TAB 
mode.  These  sentiments  are  also  reflected  in  comments  such  as: 
"At  peak,  it  was  impossible  to  evaluate  command  dispatch,"  "At 
peak  load,  I could  not  keep  up,"  and,  "At  peak,  it  was  impossible 
to  evaluate  the  command  dispatch  status." 


Another  aspect  of  the  M§S  algorithm  which  enhanced  the  con- 
trollers' feelings  of  urgency  in  keeping  the  queue  small  was  the 
fact  that  commands  timed  out  after  25  seconds.  The  controllers 
were  in  strong  agreement  with  regard  to  the  timeout  feature. 

Some  of  the  comments  made  were:  "Commands  should  blink  until  the 

controller  gets  a chance  to  activate  it  — in  no  instance  should 
it  be  erased,"  "On  all  runs  the  problem  was  the  timeout  feature. 

I believe  the  command  should  blink  until  action  is  taken,"  "Timeout 
hurried  my  dispatch  to  the  point  that  command  evaluation  suffered," 
and,  "With  the  knowledge  that  the  commands  would  time  out,  com- 
mand dispatch  was  hurried  to  the  point  that  careful  evaluation  was 
impossible."'  The  timeout  feature  was  particularly  bothersome 
for  the  exception  conditions.  Controllers  included  comments  such 
as:  "Timeout  complicates  exception  procedures"  and  "Timeout 

caused  problems  during  failures,” 


The  M§S  conflict  list  was  frequently  commented  upon.  In 
general,  the  subjects  discounted  its  utility  with  comments  such 
as:  "I  ignored  it  (conflict  list)"  and  "Sometimes  the  conflict 

list  didn't  work." 


Although  most  of  the  controllers  felt  the  3-second  gap 
between  VRS  transmissions  was  appropriate,  one  controller  indicated 
that  the  gap  length  should  be  keyed  to  the  traffic  load.  He  com- 
mented that  though  the  3-second  gap  was  "about  right  in  moderate 
traffic"  it  was  "too  long  in  crucial  situations." 

One  subject  observed  that  the  "cleared  for  approach"  command 
was  not  acknowledged  in  a clear  manner.  He  indicated  that  a 
blinking  "W"  (WILCO  acknowledgement)  was  required  in  addition  to 
the  flashing  data  block,  since  the  latter  normally  indicates  the 
aircraft  is  in  a handoff  mode.  Another  subject  indicated  that  the 
tag  clutter  on  the  base  leg  area  along  with  the  flashing  handoff- 
to-tower  was  annoying  and  complicated  the  detection  of  exception 
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9.  OBJECTIVE  DATA  ANALYSIS 


This  section  describes  the  results  of  the  analysis  of  the 
objective  (computer-generated)  measures  in  terms  of  the  statistical 
relationships  identified.  No  attempt  is  made  to  infer  the  opera- 
tional significance  of  the  findings  except  as  necessary  to  deter- 
mine the  course  of  further  analysis.  Throughout  this  section, 
key  findings  are  underlined. 

Analysis  of  the  objective  data  emphasized  the  analysis  of  the 
results  for  the  three  modes  at  the  20,  50,  and  80  percent  levels. 
This  analysis  was  divided  into  three  areas:  (1)  analysis  of  the 

dynamic  time  line  measures  recorded  for  each  message  during  the 
experiment,  (2)  analysis  of  selected  measures  from  the  end-of-run 
summary  report,  and  (3)  analysis  of  other  data  such  as  the  con- 
flict reports  or  the  number  of  missed  commands  collected  by  the 
observer  controller  or  the  pilots  during  the  course  of  the  run. 

In  a somewhat  reduced  form,  these  analyses  were  also  produced  for 
the  voice-only  runs. 


9.1  DYNAMIC  TIME  LINE  MEASURES 


As  previously  discussed,  one  of  the  major  data  sources  of 
this  experiment  was  the  measurement  of  time  intervals  for  each 
message.  From  a statistical  standpoint,  the  use  of  all  these 
measurements  increases  the  sample  size  and  thereby  the  statistical 
power  of  the  analysis  as  long  as  those  measurements  are  independent. 
This  usage  was  somewhat  complicated  by  the  incorporation  of  air- 
craft load  as  a dynamic  factor  within  each  hour's  run.  Therefore, 
each  measurement  must  be  considered  as  being  functionally  related 
to  the  actual  aircraft  load  at  the  time  the  measurement  was  made. 

As  was  learned  during  the  Data  Link  Phase  2A  experiment  (Reference 
5),  this  relationship  could  be  expressed  in  a simple  linear  form: 


7 0 


’ — — - 


a + bt 

the  output  measure 

fitted  regression  coefficients 

the  time  of  the  measurement  (a  proxy  for  load) 


Before  the  analyses  of  the  test  modes  was  started,  the  data 
from  each  experiment  was  processed  using  a simple  linear  regression 
program  (from  the  Xerox  Statistical  Subroutine  Library)  in  which 
the  following  transformation  was  made  to  all  data  to  remove  the 
time  effect: 


new  value  of  the  data  measure 


of  the  data  measure 


value  of  the  regression  equation  fitted  to  the 
original  data  and  evaluated  at  subscripted  time 
t minutes 


This  transformation  converted  the  variations  around  the  best 
linear  fit  of  the  data  to  variations  around  the  value  at  the 
30  minute  point  in  the  run  as  shown  in  Figure  13.  The  computed 
slopes,  b,  were  saved  for  subsequent  analysis. 


Three  preliminary  studies  were  conducted  to  test  the  assump- 
tions of  the  techniques  employed  in  the  following  analysis  and 
to  aid  in  the  interpretation  of  the  results  of  those  analyses. 

The  first  was  a correlation  study  to  determine  the  interrelation- 
ships among  the  two  communications  time  measures  which  were  de- 
fined for  all  modes  — message  transaction  time  and  command 
initiation  delay.  The  results  of  this  study  indicate  that  the 
two  measures  are  highly  correlated.  For  data  link  messages,  the 
measures  have  a .96  correlation  coefficient;  while  for  voice  link 


The  second  preliminary  study  was  a test  for  homogeneity  of 
variance,  a basic  assumption  underlying  analysis  of  variance  and 
the  F test,  the  procedures  to  be  used  in  the  following  analyses. 
In  this  study,  the  residual  or  error  variance  from  each  of  the 


Time 

Interval 

Measurement 


30  minute 
estimated  value 


Experiment  Time 


X = old  data  value 
O = transformed  value 


The  transformation  is  made  such  that  the  differences 
between  the  original  data  and  the  fitted  regression  line, 
d,  is  maintained  as  an  equal  difference  between  the  trans 
formed  values  and  the  estimated  value  at  30  minutes. 


NOTE 


Illustration  of  the  Transformation 
Measures  to  Eliminate  Dependency 
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linear  regression  analyses  was  compared  in  two  ways  using  the 
Bartlett  and  Cochran  tests  (Reference  6).  The  Bartlett  procedure 
tests  whether  all  the  sample  variances  can  be  considered  equal 
(homogeneous).  The  Cochran  procedure  tests  whether  the  largest 
sample  variance  is  different  from  the  others,  a common  occurrence 
which  can  significantly  affect  the  results  of  the  Analysis  of 
Variance.  These  procedures  were  performed  for  the  voice  and  data 
link  message  transaction  time  and  command  initiation  delay 
measures.  In  all  cases,  the  tests  indicated  a high  degree  of 
heterogeneity . 

The  last  preliminary  study  investigated  the  autocorrelation 
between  successive  measurements  in  a run.  Using  a procedure 
developed  by  Bartlett  (Reference  7),  the  serial  correlation  (i.e., 
the  correlation  between  successive  measurements)  was  tested  to 
determine  if  it  was  significantly  different  from  zero.  In  all 
cases,  the  serial  correlation  was  significart  and  positive,  in- 
dicating a relationship  between  successive  measurements. 

A study  of  the  correlation  between  values  of  the  residual 
variance  and  the  serial  autocorrelation  for  each  experiment  run 
indicates  that  both  the  heterogeneity  of  variance  and  the  positive 
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serial  autocorrelation  are  manifestations  of  an  incomplete  elimina- 
tion of  the  effects  of  the  changing  load.  The  effects  of  these 
factors  are  to  overstate  the  level  of  significance  in  the  Analysis 


of  Variance.  Therefore,  in  all  subsequent  analvsis,  a very  con- 


servative significance  level  (a>.99)  will  be  used  in  all  tests  of 


the  dynamic  time  line  measures, 


The  first  step  in  this  portion  of  the  analysis  consisted  of 
making  Analysis  of  Variance  runs  for  command  initiation  delay 
(CID)  and  message  transaction  time  (MTT)  for  each  type  of  aircraft 
(i.e.,  data  link  or  voice  link  equipped).  These  analyses  were 
performed  using  the  Biomedical  Computer  Program,  BMD11V,  Multi- 
variate Analysis  of  Variance  (Reference  8) . 


As  was  mentioned  previously  in  Section  6,  "Test  Design,"  the 
experimental  design  for  this  study  consisted  of  two  replications 
of  a 3 x 3 x 3 factorial  with  the  run  sequence  blocked  in 


Greco-Latin  Square  pattern.  For  this  problem,  the  model  of  the 
contributions  of  the  different  variable  effects  on  the  measured 
data  can  be  expressed  in  linear  model  form  as: 


Y (i,  j) 


+ Mi  + Pj  + rij  + Ck  + 


D*  + Sm  + * 


(3) 


where  y(i,j) 


u 


D 


a 


s 


m 

e 


the  measured  value 

the  grand  average  of  the  data 

the  effect  of  mode  i 

the  effect  of  percentage  j 

the  interaction  effect  between  mode  i and  per- 
centage j 

the  effects  of  the  sequence  in  which  the  subject 
experienced  the  modes 

the  effects  of  the  sequence  in  which  the 
percentages  were  used  within  each  mode 
the  effects  of  subject  m 
normally  distributed  error 


Throughout  this  discussion,  the  following  coding  system  was  used 

1.  For  modes  - 1 = CBA,  TAB  List,  Single  (TAB) 

2 = CBA,  FDB , Multiple  (FDB) 

3 = CBE , FDB  (CBE) 

2.  For  percentages  - 1 = 20 1 

2 = 501 

3 = 801 

3.  The  interaction  index  indicates  the  appropriate  pair  of 


the  mode  and  percentage  indices,  for  example 
interaction  effect  between  FDB  and  801. 

. t23 

= the 

4 . For 

mode  sequence  - 1 = 

The  sequence 
mode  3 

mode 

1. 

mode 

2, 

2 = 

The  sequence 
mode  1 

mode 

2, 

mode 

3, 

3 = 

The  sequence 
mode  2 

mode 

3, 

mode 

1, 
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mode  A has  the  percentage 

sequence  1 , 2,3 

mode  B has  the  percentage 

sequence  2,  3,  1 

mode  C has  the  percentage 

sequence  3,1,  2 


sequence 


2 = mode  A has  the  percentage 

sequence  2,  3,  1 

mode  B has  the  percentage 

sequence  3,  1,  2 

mode  C has  the  percentage 

sequence  1,  2,  3 

3 = mode  A has  the  percentage 

sequence  3,  1,  2 

mode  B has  the  percentage 

sequence  1 , 2 , 3 

mode  C has  the  percentage 

sequence  2,  3,  1 

6.  The  subject  index  indicates  six  different  individuals 
selected  at  random  from  the  NAFFC  pool. 

In  this  model,  the  effects  of  mode  and  percentage  and  the 
mode/percentage  interaction  are  used  to  estimate  the  value  cf  the 
measure  under  different  conditions.  The  other  effects  are  in- 
cluded to  complete  the  description  of  key  variables  and  to  account 
for  all  contributions  to  the  total  variance.  In  presentations  of 
the  calibrated  parameters,  block  and  subject  effects  are  shown  for 
completeness.  However,  in  estimating  the  value  for  a particular 
mode/percentage  combination,  they  can  be  given  an  average  of  zero 
since  they  are  of  no  operational  importance.  The  model  given  in 
(3)  was  fitted  to  the  data  for  each  measure  and  percentage  level 
using  BMD11V. 


Hypotheses  were  tested  for  each  effect,  including 

1.  Mode  effect  is  significant 

2.  Percentage  effect  is  significant, 


HBNte 


3.  Interaction  effect  is  significant, 

4.  Mode  sequence  effect  is  si;  .mi. 


3.  Percentage  sequence  effect  i : : i.  . ■ t , 

6.  Subject  effect  is  sign: fu  in t , 

In  ail  these  hypothesis  test-,  the  null  hypothesis  (H(|)  is 
that  the  individual  effects  are  equal  to  each  other  and  to  zerc, 
while  the  alternate  hypotehsis  (H.)  is  that  at  least  one  of  the 
effects  is  not  equal  to  the  others.  For  example,  for  the  mode 
effect : 


H 


0 • 


Mi  = M2  = M3  = 0 


H ^ : Mj  f M-  i j j for  at  least  one  i and  j 

The  results  of  each  analysis  of  variance  run  and  of  each  hypothesis 
test  are  as  follows: 


9.1.1  Results  of  Linear  Model  Analysis  for  Command  Initiation 
Delay  for  Data  Link  Aircraft^ 

Model:  CIDd  (i,j)  = y + Mj  + Pj  + 1^  + Ck  + D^  + Sm  + e 

where  CID^  = Command  initiation  delay  for  Data  Link  aircraft 

U = overall  average 
= effect  of  mode,  i 
Pj  = effect  of  percentage,  j 

I = interaction  effect  between  mode,  i,  and  percentage,  j 
Ck  = effect  of  mode  sequence,  k 
D^  = effect  of  percentage  sequence,  Z 
Sm  = effect  of  subject,  m 
e = normally  distributed  error 


Fstimated  from  experimental  data: 


y 


4.405  seconds 


M 

M 

M 


1 

2 

i 


-1.746 

.36.3 

1.383 


second  s 
seconds 
seconds 


s igni f icant 


214  seconds 
144  seconds 
070  seconds 


not  significant 


in  second 


403  seconds 
642  seconds 
239  seconds 


298  seconds 
710  seconds 
412  seconds 


.573  seconds 
-.918  seconds 
1.220  seconds 
.782  seconds 
.234  seconds 
.549  seconds 


Where  C = average  mode  sequence  effect  = 0 

D = average  percentage  sequence  effect 
S = average  subject  effect  = 0 
e = average  error  = 0 


\ j 

1 

3 

i 

-.161 

-.797 

.958 

2 

- .027 

.921 

- .894 

3 

.188 

-.124 

-.064 

9.1.2  Results  of  Linear  Model  Analysis  for  Message  Transaction 
Time  for  Data  Link  Aircraft 


iru* 

S-.  ^ 


r-i 


Model:  MTTd  (i,j)  = u + Mj  + ?■  + Ij.  + ck  + Ds>  + sm  + e 

where  MTT^  = Message  transaction  time  for  Data  Link  aircraft 

y = overall  average 
M = effect  of  mode,  i 
Pj  = effect  of  percentage,  j 

[^.  = interaction  effect  between  mode,  i,  and  percentage  j 
= effect  of  mode  sequence,  k 
Dp  _ effect  of  percentage  sequence,  l 
S„  = effect  of  subject,  m 
e = normally  distributed  error 

Estimated  from  experimental  data: 

p = 8.429  seconds 


8.429  seconds 
= -1.727  seconds 
.475  seconds 


= 1.252  seconds 


.149  seconds 
.141  seconds 
.008  seconds 


s i gni f ic  an  t 


not  significant 


I . . in  seconds 
ij 


1 

2 

3 

1 

-.313 

-.881 

1.194 

2 

. 201 

1.034 

-1.235 

3 

.112 

-.153 

.041 

.213  seconds 
.610  seconds 
,397  seconds 

.450  seconds 
.660  seconds 
.210  seconds 


signi f icant 


significant 


signi ficant 
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s igni f ic  ant 


E 


S2 


S5  = 


S6  = 


.348  seconds 
-.817  seconds 
-.840  seconds 
.567  seconds 
.359  seconds 
.383  seconds 


For  example: 


MTTd  (3,3) 


p+Mj  + Pj  + Ij^  + C + D + S + e 


= 8.429  + 1.252  + .00  + .041  = 9.722  seconds 


9.1.3  Results  of  Linear  Model  Analysis  for  Command  Initiation 
Delay  for  Voice  Link  Aircraft 


Model 

where 


: CIDV  (i.j)  = p ♦ Mi  ♦ Pj  ^ I±j  + Ck  + ♦ Sm  ♦ e 


CID,,  = Command  initiation  delay  for  Voice  Link  aircraft 


0 = overall  average 
= effect  of  mode,  i 


Pj  = effect  of  percentage,  j 

1.^  = interaction  effect  between  mode,  i,  and  percentage,  j 
C,^  = effect  of  mode  sequence,  k 


= effect  of  percentage  sequence,  l 


S = effect  of  subject,  m 
m J ’ 


e = normally  distributed  error 
Estimated  from  experimental  data: 


u 

* 

4.327 

second  s 

M1 

= 

-1.341 

seconds 

M2 

= 

-.111 

seconds 

*3 

= 

1.452 

seconds 

P1 

= 

- . 245 

seconds 

P2 

= 

-.14  3 

seconds 

= 

.388 

seconds 

s igni f icant 


sigm  icant 
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I . . in  seconds 
ij 


.047  seconds 
,091  seconds 
.138  seconds 

.030  seconds 
.232  seconds 
.202  seconds 

,323  seconds 
.090  seconds 
.638  seconds 
.379  seconds 
.019  seconds 
.045  seconds 


significant 


not  significant 


significant 


s igni f icant 


For  example: 


CIDy  (2,1)  = v + M2  + P1  + I21  +C+D+S+e 

= 4.327  - .111  - .245  - .048  = 3.923  seconds 


9.1.4  Results  of  Linear  Model  Analysis  for  Message  Transaction 
Time  for  Voice  Link  Aircraft 

Model:  MTTy  (i,j)  = y + Mi  + Pj  + I — +Ck  + D?,  + Sm  + e 

where  MTTy  = Message  transaction  time  for  Voice  Link  aircraft 

y = overall  average 
M^  = effect  of  mode,  i 
Pj  = effect  of  percentage,  j 

I . j = interaction  effect  between  mode,  i,  and  percentage,  j 
= effect  of  mode  sequence,  k 


I III  liltiwil  ■*  — «'  — * m*-'.  ■ 


I 


I ' 


= effect  of  percentage  sequence,  l 
Sm  = effect  of  subject,  m 
e = normally  distributed  error 

Estimated  from  experimental  data: 

U = 9.010  seconds 


Mj  = -1.290  seconds 
M2  = —.298  seconds 
Mj  = 1.588  seconds 

Pj  = .250  seconds 
P2  = -.122  seconds 
P,  = ^.128  seconds 


I . . in  seconds 
ij 


1 

2 

3 

1 

- .122 

-.39  3 

. 515 

2 

.036 

.0  36 

-.072 

3 

.086 

.357 

- .443 

C,  = 


n?  - 

D,  = 


S3  = 

s»  - 


.116  seconds 
-.028  seconds  > 
-.088  seconds  \ 

- .066  seconds  / 
.564  seconds  > 

- .498  seconds  \ 

- .072  seconds 
-.053  seconds 
-.087  seconds 
-.016  seconds 

.011  seconds 
.217  seconds 


For  example: 


MTTy  (3,2) 


= p + Mj  + P2  + I3  2 + 

= 9.010  + 1.588  - 1 00 


significant 


not  significant 


not  significant 


not  significant 


significant 


not  significant 


C + D + S + e 
+ .00  = 10.598  seconds 
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Examination  of  the  above  data  shows  that  the  average  CID 
for  voice  and  data  link  aircraft  were  approximately  equal  at 
slightly  under  4.5  seconds.  For  MTT,  messages  to  voice-link 
aircraft,  with  an  average  of  9.0  seconds,  were  somewhat  longer 
than  messages  to  data  link  aircraft,  with  an  8.4  second  average. 

The  mode  effect  was  always  significant.  The  TAB  mode  was 
fastest  and  was  approximately  2 seconds  better  than  the  FDB  mode, 
which  in  turn  was  approximately  1 second  better  than  the  CBE  mode 
in  all  four  cases.  The  percentage  effect  was  statistically 
significant  only  in  the  case  of  voice  link  aircraft  for  CID,  and 
even  then  the  range  oF  difference  was  only  slightly  higher  than 
.5  second.  The  interaction  effect  was  generally  significant  and 
demonstrated  the  same  pattern  for  all  four  measures,  as  shown  in 
Figure  14,  although  the  pattern  was  somewhat  stronger  for  the  CID 
measures  than  for  the  MTT  measures.  For  the  TAB  mode,  the  50 
percent  level  was  the  fastest  and  the  80  percent  the  slowest. 

The  converse  was  true  for  the  FDB  mode  in  which  the  50  percent 
level  was  slowest  and  the  80  percent  fastest.  For  the  CBE  mode, 
all  percentage  levels  were  approximately  equal. 

To  complete  the  description  of  the  dynamic  data,  the  slope 
values,  b,  computed  in  the  preliminary  regression  step  (1)  were 
analyzed  to  determine  if  the  relationship  with  load  was  in  any  way 
affected  by  the  experiment  variables.  These  values  were  analyzed 
using  BMD11V  to  fit  a model  with  the  same  form  as  shown  in  (3). 

Since  only  one  value  was  computed  per  run  and  the  homogeneity  of 
variance  is  not  a problem,  the  tests  of  hypotheses  used  a re- 
jection level  of  .05.  As  shown  in  Table  7,  there  were  no  signifi- 
cant effects  for  any  of  the  measures.  The  magnitude  of  the  slopes 
indicate  that  the  MTT  values  were  more  significantly  affected  by 
increasing  load,  particularly  the  MTT  for  voice  link  aircraft,  which 
increased  over  five  times  as  fast  as  any  other  measure. 

The  results  of  the  slope  analysis  can  be  combined  with  the 
analysis  of  the  measures  to  produce  point  estimates  of  the  measures 
at  different  loads  by  the  use  of  the  model: 

y(i,j)  = p + Hi  ♦ V-  * I ♦ bi;j  * (t- 30)  (4) 
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where  Y(i,j)  = the  measure  of  interest 

y = average  effect  for  that  measure 
= effect  of  mode  i for  that  measure 
Pj  = effect  of  percentage  j for  that  measure 
b = slope  for  measure  of  interest 
t = experiment  time  at  which  load  equals  desired 
value  (See  Section  5.5,  "Traffic  Samples") 

For  example,  if  the  value  of  Command  Initiation  Delay  is  de- 
sired for  voice  link  aircraft  for  mode  one  (TAB)  at  percentage 
level  two  (50  percent)  at  a load  of  approximately  seven  aircraft 
(i.e.  , about  4 2 minutes  into  the  run) , then  that  value  can  be 
computed  from  the  values  y,  N^,  , and  1^  from  section  9.1.3  and 

the  value  of  b^^  from  Table  7 as  follows: 

C I D v ( 1 , 2 ) = y + + P.  + Ii;j  + btj  * (t-30) 

= 4.327  - 1.341  - .143  - .500  + .00857  * (42-30) 

= 2.446  seconds 

9.2  END  OF  RUN  SUMMARY  MEASURES 

For  each  run,  a summary  report  (shown  in  Figure  15)  was 
prepared.  Twenty-one  measures  (circled  items)  were  retrieved  from 
each  report  and  analyzed  using  analysis  of  variance  techniques. 

In  order  to  understand  the  interrelationships  of  the  measures, 
a correlation  study  was  performed.  The  results  of  that  study  are 
shown  in  Table  8. 

The  data  in  Table  8 was  analyzed  using  Correlation  Cluster 
Analysis  (Reference  9)  , a graphic  technique  for  determining  groups 
of  measures  which  are  highly  intercorrelated . This  analysis  showed 
that  the  21  measures  are  fairly  independent.  Only  13  of  the 
measures  could  be  combined  into  4 groups:  average  queue  size, 

average  MTT(VL),  average  MTT(DL),  average  CID(VL) , and  average 
CID(DL).  All  were  highly  correlated,  having  a minimum  correlation 
between  pairs  of  measures  of  .777.  The  average  queue  delay  voice 
link,  QD(VL) , average  queue  delay  data  link,  QD(DL) , average  service 
time  voice  link,  ST(VL),  and  average  service  time  data  link,  ST(DL) , 
are  correlated  at  a minimum  of  .624.  Two  other  pairs  of  measures, 
total  MTT(VL)  and  max  MTT(VL)  at  .800,  and  the  number  of  VRS  halt/ 
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Figure  15.  Data  Reduction  Program  Summary  Printout 


RESULTS  OF  CORRELATION  STUDY  FOR  END-OF-RUN  SUMMARY  MEASURES 


resume  sequences  and  the  total  time  the  VRS  was  halted  at  .738, 
could  also  be  grouped.  The  other  measures  could  all  be  considered 
as  independent. 


These  groupings  reflect  the  fact  that  the  measures  are 
strongly  related  and  that  each  measure  cannot  be  considered  a 
separate  indicator  of  system  value.  The  evaluation  of  these 
measures  should  be  considered  in  these  groups. 

Each  measure  was  analyzed  using  BMD11V.  A model  of  the 
form  of  (3)  was  fitted  to  each  measure  and  the  significance  of 
the  parameters  was  tested  at  the  .05  level.  The  results  of  these 
analyses  are  summarized  in  Table  9. 

Study  of  the  group  1 measures,  average  queue  size,  average 
MTT(VL)  , average  MTT(DL),  average  CID(VL),  and  average  CID(DL) , 
indicates  very  similar  responses,  as  expected.  In  all  cases  the 
mode  effect  is  significant,  with  the  TAB  mode  having  the  lowest 
values  followed  by  the  FDB  mode  and  with  CBE  having  the  largest 
values.  The  percentage  effect  is  significant  only  for  the 
average  MTT(VL)  measure,  with  that  value  decreasing  nearly 
linearly  for  increasing  percentage. 

The  group  2 measures,  average  QD(VL),  average  OD(DL),  average 
ST(VL),  and  average  ST(DL),  indicate  significant  values  for  the 
mode  effects;  however,  since  QD  and  ST  are  only  defined  for  the 
TAB  mode,  the  results  should  be  interpreted  as  average  values  of 
.462  seconds  for  average  QD(VL),  .543  seconds  for  average  QD(DL) , 
7.75  seconds  for  average  ST(VL) , and  2.17  seconds  for  average 
ST(DL).  No  other  significant  effects  were  observed. 

Group  3 measures,  total  MTT(VL)  and  maximum  MTT(VL) , although 
highly  correlated  have  some  differences  in  the  significance  of  the 
effects.  For  total  MTT(VL),  both  the  mode  and  percentage  effects 
are  significant  with  the  modes  ranked  in  the  normal  TAB,  FDB,  CBF. 
sequence.  The  total  MTT(VL)  decreases  with  increasing  percentage 
of  data  link  aircraft  as  expected.  For  the  maximum  MTT(VL) , 
however,  there  is  no  significant  mode  effect,  although  the  percentage 
effect  also  decreases  with  increasing  data  link  percentage. 
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TABLE  9.  RESULTS  OF  ANALYSIS  OF  VARIANCE  FOR  END-OF-RUN  SUMMARY  DATA 


Not  Significant 


N't  Significant 


For  data  link  aircraft,  the  two  complementary  measures, 
total  MTT(DL)  and  maximum  MTT(DL),  show  somewhat  different  re- 
sponses. The  total  MTT(DL)  also  has  significant  mode  and  per- 
centage effects.  The  mode  effects  are  in  the  normal  relation- 
ship, TAB,  FDB,  CBE.  The  percentage  effect,  however,  increases 
with  increasing  data  link  percentage  as  expected.  For  the 
maximum  MTT(DL)  measure  only  the  mode  effect  is  significant. 

In  this  case,  the  CBE  mode  has  the  lowest  maximum  value,  the 
TAB  mode  the  second  lowest,  and  FDB  the  worst. 

Among  the  other  measures,  the  only  one  with  any  significant 
effect  is  the  average  error  in  arrival  time  which  has  a signifi- 
cant mode  effect. 

9.3  OTHER  MEASURES 

As  described  in  Section  5,  "Test  Conduct,"  additional  infor- 
mation was  collected  by  the  observer  controller  and  the  simula- 
tion pilots  during  the  course  of  each  run.  Specifically,  the 
observer  controller  recorded  warnings  of  imminent  conflicts  as 
detected  by  the  subject.  This  information  included  the 
identification  of  the  aircraft  pair  in  conflict  and  the  time  the 
subject  issued  the  warning. 

In  addition,  the  pilots  kept  a handwritten  tally  of  the 
number  of  non-M^S  commands  issued  by  the  subject.  The  tabula- 
tions of  such  controller  generated  commands  by  mode  and  per- 
centage are  given  in  Table  10.  As  can  be  seen,  there  is  no 
pattern  to  the  responses. 

TABLE  10.  TABULATION  OF  CONTROLLER  GENERATED  COMMANDS 


The  subject  controllers  were  instructed  to  inform  the 
observer  whenever  it  was  necessary  to  maneuver  the  aircraft  to 
provide  separation.  The  observer  recorded  the  aircraft  indenti- 
fications  and  the  time  at  which  the  warning  was  given.  After 
the  simulation  run,  an  analysis  of  the  aircraft  tracks  was 
performed  to  determine  which  pairs  of  aircraft  actually  came 
into  conflict  and  the  time  at  which  those  conflicts  began. 

The  two  sets  of  data  were  compared  manually,  and  counts  were 
made  of  the  number  of  times  in  which  (1)  the  controller  detected 
a conflict  situation  which  did  not  materialize,  (2)  an  actual 
conflict  was  not  detected  by  the  subject,  and  (3)  the  controller 
detected  the  conflict  more  than  15  seconds  after  it  started. 

The  results  of  this  effort  are  given  in  Table  11.  The  missed 
conflicts  were  further  examined  for  some  indication  of  severity. 
Several  of  the  actual  conflicts  were  very  minor  incidents  in  which 
the  aircraft  were  in  conflict  for  only  one  scan.  If  these  con- 
flicts are  removed  from  consideration,  the  numbers  of  missed 
conflicts  are  reduced  to  those  indicated  in  parentheses  in 
Table  11. 

Statistical  analysis  of  the  conflict  data  is  somewhat 
complicated,  since  they  are  discrete,  relatively  rare  events, 
and  since  the  number  of  actual  conflicts  in  any  of  the  simula- 
tion experiments  was  not  controlled  but  was  itself  a random 
variable.  Assuming  that  the  number  of  actual  conflicts,  n,  in 
any  hour  is  a Poisson  variable  with  unknown  arrival  (occurrence) 
rate,  X,  then 


Prob  (n)  = e . 

If  the  detection  of  a conflict  is  a Bernoulli  event  with  the 
probability  of  failing  to  detect  the  conflict,  6,  the  number  of 
missed  detections,  x,  given  the  number  of  actual  conflicts,  n, 
is  then  distributed  as  a binomial  with 


Prob  ( 


X|n)  " (x) 


ex  (i-e)n'x 


I 


TABLE  11.  REDUCED  CONFLICT  DATA 


MODE 

20% 

50% 

80% 

TOTAL 



E 

M 

m 

E 

M 

a 

E 

M 

n 

E 

M 

B 

FDB 

5 

2(2) 

3 

2 

4(1) 

4 

4 

6(3) 

3 

11 

12(6) 

10 

TAB 

5 

5(1) 

5 

0 

2 

4 

4 

2 

4 

9 

9(5) 

13 

CBE 

3 

2(1) 

2 

2 

2(1) 

5 

2 

0 

1 

I 

4(2) 

8 

TOTAL 

13 

9(4) 

10 

D 

8(2) 

13 

10 

8(5) 

8 

27 

25(13) 

31 

E = Conflict  detected  by  controller  which  did  not  materialize 

M = Actual  conflict  which  was  not  detected  by  the  controller 

L = Actual  conflict  which  was  detected  at  least  15  seconds  after  the 
start  of  the  conflict 
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- 


Prob  (z) 


e'Xz(xe) z xa-e) 


which  is  the  expression  for  the  probability  of  Poisson  variable 
with  arrival  rate  = X0. 

In  addition,  since  the  observed  average  conflict  rates  were 
small,  it  is  not  valid  to  approximate  the  Poisson  data  by  a normal 
probability  distribution.  Therefore,  classical  statistical  test 
procedures  which  assume  that  the  variables  are  normally  dis- 
tributed are  invalid. 


Several  alternate  procedures  were  applied  to  the  missed 

conflict  data,  both  filtered  and  nonfiltered,  to  determine  whether 

the  data  for  the  three  modes  exhibited  significant  differences. 

2 

A x test  statistic  assuming  Poisson  distributed  variables  and 
Friedman's  nonparametric  two-way  analysis  of  variance  both  in- 
dicated that  the  effect  due  to  mode  was  not  significant  for  the 
missed  conflicts  in  both  the  filtered  and  nonfiltered  data. 

9.4  ANALYSIS  OF  VOICE  RUN  DATA 

Voice  runs  for  the  TAB  and  FDB  modes  were  added  to  the  ex- 
periment to  provide  a baseline  against  which  to  compare  the  ex- 
periment results.  The  adjustment  for  load  was  performed  on  the 
dynamic  measures  for  the  voice  runs  as  for  the  VRS  runs.  The 
dynamic  measures  were  analyzed  using  BMD11V  to  fit  a model  of 
the  form: 


y(i)  =y+Mi+Sm+e 

where  y(i)  = the  dynamic  measurement 
p = overall  average 
Mi  = effect  of  mode  i 
Sm  = effect  of  subject  m 
e = normally  distributed  error 

The  results  of  this  analysis  for  the  CID  measures  are  shown 
in  Sections  9.4.1  and  9.4.2.  It  can  be  seen  that  the  mode 
effect  is  not  significant.  The  CID  values  are  approximately 
equal  to  the  TAB  CID  values  for  the  mixed  voice  and  data  link  runs. 
The  MTT  values  represent  a 1 second  improvement  for  TAB  and  a 2 
second  improvement  for  FDB. 

9.4.1  Results  of  Linear  Model  Analysis  for  Command  Initiation 
Delay  for  Voice  Runs 

Model:  CIDy(i)  = y + M^  + Sm  + e 

where  CIDy(i)  - measured  command  initiation  delay 
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= overall  mean 
= effect  of  mode  i 
= effect  of  subject  m 
= normally  distributed  error 

= 3.076  seconds 


.012  seconds 
.012  seconds 


.134  seconds 
-.702  seconds 
.122  seconds 
.573  seconds 
.174  seconds 
-.301  seconds 


not  significant 


not  significant 


For  example: 


CIDy(l)  “p+Mj+S+e 

= 3.076  - 0.043  = 3.076  seconds 


9.4.2  Results  of  Analysis  of  Variance  for  Message  Transaction 
Time  for  Voice  Runs 


Model:  MTT,,(i)  = p + M-  + S + e 

v i m 

where:  MTTv(i)  = measured  message  transaction  time 


y = overall  mean 
Mj,  = effect  of  mode  i 
Sm  = effect  of  subject  m 
e = normally  distributed  error 


p = 6.665  seconds 


= .035  seconds 

M2  = -.035  seconds 


not  significant 


S1  = 

. 234 

seconds 

II 

c r 

-.790 

seconds 

S3  " 

.369 

seconds 

S4  = 

.731 

seconds 

S5  = 

.162 

seconds 

s*  = 

-.706 

seconds 

s igni ficant 


For  example:  MTTy(2)  =p+M2+5+e 

= 6.665  - 0.035  = 6.665  seconds 


With  only  12  data  points  for  each  measures,  the  slopes,  the 
end-of-run  summary  data,  and  the  other  measures  were  not 
statistically  analyzed.  The  average  values  for  each  mode  were 
computed  and  are  shown  in  Table  12.  For  the  end-of-run  summary 
data,  except  for  the  number  of  tag  repositionings,  the  two 
modes  were  virtually  the  same.  The  FOB  mode  had  many  more  tag 
repositionings  than  TAB.  In  terms  of  the  other  measures,  the 
TAB  mode  had  less  missed  commands  and  controller  generated 
clearances  than  FDB.  However,  FDB  had  less  than  half  the  extra 
and  missed  conflict  warnings. 

When  compared  to  the  mixed  data  link  and  voice  link  runs 
shown  in  Tables  9,  10,  and  11,  most  of  the  results  were  reasonably 
in  agreement.  The  only  meaningful  differences  were:  (1)  the 

total  and  average  MTT  values  are  substantially  lower  for  the 
voice-only  runs,  and  (2)  in  all  cases,  most  of  the  other  measures, 
controller  clearnaces,  extra  warnings,  and  missed  warnings  were 
somewhat  lower  for  the  voice-only  runs. 


TABLE  12.  AVERAGE  RUN  VALUES  BY  MODE  FOR  VOICE  RUNS 


END 

-OF -RUN  SUMMARY  MEASURES 

TAB  LIST 

FULL  DATA  BLOCK 

1. 

Max  Queue  Size 

4.0 

3.8 

2. 

Avg  Queue  Size 

.338 

.366 

3. 

Max  Aircraft  Load 

9.5 

9.8 

4. 

Total  MTT  (VL) 

1717.2 

1643.7 

5. 

Total  MTT  (DL) 

- 

- 

6. 

Avg  MTT  (VL) 

7.27 

7.05 

7. 

Avg  MTT  (DL) 

- 

8. 

Max  MTT  (VL) 

22.4 

21.9 

9. 

Max  MTT  (DL) 

- 

- 

10. 

Avg  CID  (VL) 

3.57 

3.62 

11. 

Avg  CID  (DL) 

- 

- 

12. 

Avg  QD  (VL) 

3.57 

undefined 

13. 

Avg  QD  (DL) 

- 

- 

14  . 

Avg  ST  (VL) 

4 . 58 

undefined 

15. 

Avg  ST  (DL) 

- 

- 

16. 

Avg  Error  in  Arrival  Time 

.88 

-.28 

17. 

Std.  Dev.  Arrival  Error 

5.47 

5.55 

18. 

No.  VRS  Seouences 

- 

- 

19. 

Total  Time  Halted 

- 

- 

20. 

No.  Tag  Repositionings 

1.00 

6.12 

21  . 

No.  Time  Conflicts 

.00 

.33 

OTHER  MEASURES 

1 . 

Controller  Clearances 

.83 

1.09 

2. 

Extra  Warnings 

.42 

.17 

3. 

Missed  Warnings 

.42 

.17 

4 . 

Late  Warnings 

.25 

.25 

10,  ANALYSIS  OF  RESULTS 


In  this  section,  the  most  pertinent  subjective  and  objective 
data  are  evaluated  and  reconciled  in  an  effort  to  ascertain  mode 
rankings.  Data  pertaining  to  the  following  experimental  measures 
were  used  in  this  analysis: 

1.  Controller  opinion  questionnaires 

2.  Time  to  execute  equivalent  functions  such  as  Message 
Transaction  Time  (MTT)  and  Command  Initiation  Delay  (CID) 

3.  Queue  behavior  (average  number  of  commands  awaiting 
dispatch) 

4.  Counts  of  conflicts  (separation  violations  that 
occurred  and  the  number  detected  by  controller) 

There  was  a very  definite  recurring  pattern  in  the  answers 
to  the  Post-Experiment  questionnaire.  In  a vast  majority  of  the 
questions,  the  order  of  controller  preference  with  respect  to  the 
control  modes  was  CBE/FDB/VRS , CBA/TAB/VRS , and  CBA/FDB/VRS.  This 
was  especially  evident  in  such  questions  as  A-l.A  and  A-l.B,  which 
exhibited  significant  mode  effects.  A-l.A  asked  the  subject  how 
he  believed  "this  mode  of  displaying  and  transmitting  commands 
would  affect  the  current  operational  system  for  the  same  traffic 
situation."  Question  A-l.B  asked  the  subject  to  rate  the  mode  with 
respect  to  workload,  capacity,  blunder,  and  stress.  By  assigning 
the  weight  ( + 2,  +1,  ...,  -2)  to  every  answer  to  these  questions, 
summing  the  results  algebraically,  and  dividing  the  sum  by  the 
number  of  responses,  a normalized  overall  index  of  merit  was 
computed  for  each  mode.  Table  13  lists  the  figures  used  in  de- 
riving an  overall  index  of  merit  for  each  mode  and  the  ranking 
of  each  mode  on  the  basis  of  the  index.  Not  surprisingly,  the 
rankings  in  Table  13  coincide  with  those  derived  more  straight- 
forwardly in  the  Subjective  Analysis  Section. 
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TABLE  13. 


MODE  RANKINGS  BASED  ON  POST- EXPERIMENT  QUESTIONS 
A.l-A  AND  A.l-B 


MODE 

MIX 

> 

SUM  OF  WEIGHTED 

' REPLIES 
> 

OVERALL  NORMALIZED 
INDEX 

(SUMS  V-  18) 

A.l-A 

RANK 

SUM  OF 

WEIGHTED 

REPLIES 

A.l-B 

OVERALL  NORMALIZED 
INDEX 

(SUMS  4-  24) 

A-l.B  RANK 

WORKLOAD 

CAPACITY 

BLUNDERS 

STRESS 

20% 

50% 

80% 

CBA/ 

201 

-5 

-5 

-4 

-7 

-7 

-.96 

3 

FDB 

SO* 

-5 

- .83 

3 

-7 

-5 

-6 

-5 

- .96 

3 

80% 

-5 

-8 

-6 

-7 

-9 

-1.3 

3 

CBA/ 

20% 

0 

-1 

-2 

-5 

-8 

-.67 

2 

TAB 

50% 

-3 

- . 55 

2 

-1 

1 

-5 

-6 

- .46 

2 

80% 

-7 

0 

0 

-5 

-6 

- .46 

2 

CBE/ 

20% 

3 

5 

4 

0 

-1 

+ .33 

1 

FDB 

50% 

2 

- .44 

1 

3 

5 

0 

0 

+ .33 

1 

80% 

2 

3 

3 

0 

-1 

+ .29 

1 

The  index  of  merit  ranking  derived  from  A-l.A  and  A-l.B  data 
was  verified  by  repeating  the  procedure  with  responses  to  Summary 
Questionnaire  Question  1,  which  asked  the  controllers  to  rank  all  of 
the  modes  with  respect  to  workload,  capacity,  susceptibility  to 
error,  stressfulness,  and  overall  performance.  The  calculation  of 
an  overall  rank  index  for  each  mode  under  these  five  criteria  is 
summarized  in  Table  14  on  the  basis  of  responses  shown  in  Table  6 
in  Section  7 of  this  report. 

All  the  ranking  data  presented  thus  far  will  now  be  examined 
to  extract  information  on: 

a.  Overall  ratings 
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t b.  Ratings  in  each  of  the  categories  (workload,  capacity, 

blunders,  stress) 

c.  Variations  of  ratings  with  percentage  mix. 

First,  the  rankings  obtained  from  question  A-l.A  and  the  overall 
rankings  obtained  from  Summary  Questionnaire  Question  1 are  re- 
stated in  order  in  Table  15. 


I 

TABLE  14.  OVERALL  RANKINGS  BASED  ON  SUMMARY  QUESTION  1 


't 
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MODE 

SUMMARY 
QUESTION  1 
COMPONENT 
RESPONSES* 

OVERALL  NORMALIZED 
RANK  INDEX  (SUM  4-24) 

RANK 

WORKLOAD 

CAPACITY 

BLUNDER 

co 

CO 

w 

E-. 

CO 

CBA/FDB 

13 

13 

13 

14 

2.2 

3 

CBA/TAB 

13 

13 

11 

11 

2 

2 

CBE/FDB 

10 

10 

12 

11 

1.8 

1 

o « 


o 

ID  •-< 
Z H 
M LO 
H W 
< HD 
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j >- 
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11 

14 

11 


"The  larger  the  number,  the  lower  the  rating. 
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.46 
. 58 
.46 


TABLE  15. 


SUMMARY  TABLE  OF  OVERALL  MODE  RANKINGS 


RANK 


Since  the  rankings  from  Questions  A.l.A  and  A.l.B  are  identical 
with  the  rankings  derived  from  the  components  of  Summary  Question  1 
(the  latter  representing  a direct  evaluation  of  the  modes  relative 
to  each  other) , and  since  the  Summary  Questionnaire  responses  were 
obtained  after  each  subject  had  been  exposed  to  the  three  modes, 
the  following  ranking  will  be  regarded  as  the  final  consensus  on 
the  overall  order  of  merit: 

1.  CBE/FDB 

2.  CBA/TAB 

3.  CBA/FDB 

In  arriving  at  the  final  ranking  above,  the  difference  between 
the  component  and  overall  rankings  in  Summary  Question  1 was  not 
considered  to  be  of  sufficient  weight  to  effect  a change  in  the 
final  ranking.  However,  one  explanation  for  the  difference  might 
be  that  when  faced  with  a general  question  of  choice,  the  control- 
lers expressed  their  preference  to  having  the  command  displayed  in 
the  third  line  of  the  data  block  - i.e.,  trading  off  lesser  work- 
load for  better  command  location. 

The  results  of  the  Phase  2A  DSF  experiment  indicated  that  the 
subject  controllers  distinctly  preferred  computer-generated  voice 
when  voice  link  traffic  is  heavy.  In  this  experiment,  however,  the 
subject  controllers,  while  acknowledging  that  it  was  less  tiring 
than  voice  commands,  found  the  inability  to  repeat  commands  and  the 
talking  rate  unsatisfactory.  Contrary  to  the  assumption  that 
the  controller  can  look  ahead  to  the  next  command  while  listening 
to  the  computer  - generated  voice,  the  controllers  in  this  experiment 
did  not  indicate  a predilection  to  do  so.  The  less  than  positive 
reaction  to  the  VRS  was  obtained  despite  the  fact  that  the  con- 
trollers enjoyed  the  convenience  of  employing  the  same  keyboard 
functions  to  dispatch  both  data  link  and  voice  link  commands. 

The  consistently  high  ranking  of  the  CBE/FDB  mode  indicates 
a strong  preference  for  command  location  in  the  full  data  block  as 
opposed  to  the  tabular  list,  despite  the  fact  that  CBA/FDB  is  con- 
sistently ranked  last.  The  latter  is  obviously  attributable  to  the 
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fact  that  the  controllers  intensely  dislike  the  trackball  slew 
action  required  for  each  dispatch  action.  The  principal  advantage 
of  FDB,  whether  it  be  CBA  or  CBE,  is  that  the  commands  are  dis- 
played in  each  alphanumeric  tag,  close  to  the  aircraft  in  question. 
Correlation  of  the  command  with  the  aircraft's  current  status  is 
facilitated  by  displaying  the  command  in  the  data  block.  However, 
the  controllers  were  bothered  by  having  to  prevent  overlapping 
data  blocks  which  obscure  commands.  This  was  more  significant  in 
CBA/FDB  than  CBE/FDB,  probably  because  command  display  time  was 
longer.  Unfavorable  reaction  to  CBA/TAB  stems  from  the  fact  that 
the  controllers  had  to  split  their  attention  between  the  list  and 
the  traffic  situation.  The  greater  ease  of  dispatch  — i.e.,  single 
key  depression  without  trackball  slew  — and  the  correlation  aid  of 
having  the  controller  symbol  blink  with  an  associated  blinking 
command  were  not  sufficient  to  overcome  the  unfavorable  reaction 
to  a list  format. 

The  most  significant  aspect  of  the  mode  rankings  is  the  con- 
sistently favorable  reaction  to  control-by-exception.  As  shown 
in  Table  16,  this  mode  also  ranked  first  with  respect  to  all  the 
components  of  Summary  Question  1 with  the  exception  of  Suscepti- 
bility to  Blunders;  there  it  ranked  second  in  a close  ranking  of 
all  modes.  Some  concern  was  expressed  at  not  having  pretransmission 
approval,  but  this  was  attributed  to  controller  lack  of  confidence 
in  the  M§S  algorithm  generating  conflict-free  commands.  Only  one 
controller  stated  that  he  could  not  monitor  the  commands  at  peak 
load;  another  suggested  that,  at  the  80%  traffic  mix,  the  VRS 
also  enunciate  data  link  commands  as  an  "attention  director." 

The  quantitative  measurements  made  after  the  test  runs  were 
conducted  to  provide  a basis  for  understanding  the  subjective  data. 
The  Message  Transaction  Time  (MTT)  represents  the  total  time  re- 
quired for  the  system  to  process  a given  command  and  is  thus  the 
most  important  measure  of  control  mode  performance.  Since  per- 
formance at  maximum  load  is  of  primary  interest,  the  MTT  values 
corresponding  to  that  point  in  the  test  run  (Tq  + 60  minutes) 
were  computed  using  the  regression  analysis  results.  Table  17 
contains  the  MTT  of  the  three  control  modes  for  both  data  link  and 

104 


UES£jXi££r^...'  . 


• . 

iMUi 


TABLE  16.  MODE  RANKING  BY  SUMMARY  QUESTION  1 COMPONENTS 


MODES 


RANK 


WORKLOAD 


CAPACITY 


BLUNDERS 


STRESS 


CBA/TAB 


CBE/FDB 


CBE/FDB 


CBE/FDB 


CBA/FDB 


CBA/FDB 


MODE 


DATA  LINK 


VOICE  LINK 


8.95  + (30  X .01283) 
6.24  + (30  X .01283) 
9.644  + (30  X .01283) 


8.99  + (30  X .06309) 
7.85  + (30  X .06309) 
10.93  + (30  X .06309) 


8.63  + (30  X .06309) 
7.21  + (30  X .06309) 
10.83  + (30  X .06309) 


CBA/FDB 

CBA/TAB 

CBE/FDB 


7.68  (30  X .01283) 
7.90  (30  X .01283) 
9.73  (30  X .01283) 


8.51  + (30  X .06309) 
8.11  + (30  X .06309) 
10.03  + (30  X .06309) 


CBA/FDB 

10.08  + (30  X .01283) 

CBA/TAB 

5.96  + (30  X .01283) 

CBE/FDB 

9.67  + (30  X .01283) 

voice  data  link  messages  at  the  three  traffic  mixes.  In  Table  18 
the  modes  have  been  ranked  in  order  of  increasing  MTT. 

The  ranking  of  the  modes  by  MTT  values  at  maximum  load  does 
not  conform  with  the  ranking  based  on  controller  opinion.  For 
example,  Control-by-Exception/Full  Data  Block  almost  always  ranks 
third  In  MTT,  and  consistently  ranks  first  according  to  question- 
naire results.  Control-by-Approval/Tabular  List  ranks  first  in 
MTT,  with  the  exception  of  the  801  data  link  case,  but  consistently 
ranks  second  in  controller  opinion.  Control-by-approval  full  data 
block  fares  better  in  MTT  ranking  than  in  controller  opinion, 
where  it  always  ranked  last. 

The  MTT  rankings  of  the  modes  may  not  be  as  much  at  variance 
with  the  questionnaire  data  as  would  appear  upon  examination  of 
the  command  initiation  delay  (CID)  values  obtained  by  regression 
analysis  for  the  three  modes.  The  Command  Initiation  Delay  is 
the  time  between  initial  command  display  and  the  dispatch  action 
and,  as  such,  is  a major  component  of  the  total  message  trans- 
action time.  Table  19  contains  the  CID  of  the  three  control 
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TABLE  19.  CID  AT  MAXIMUM  LOAD  FOR  DATA  LINK  AND  VOICE  LINK 


MOLE 


DATA  LINK 


VOICE  LINK 


CBA/FDB 

CBA/TAB 

CBE/FDB 


4.53  + (30  X .00831) 

2.03  + (30  X .00831) 

6.04  + (30  X .00831) 


3.92  + (30  X .00857) 
2.54  + (30  X .00857) 
5.79  + (30  X .00857) 


ICBA/FD3 


CBA/TAB 

CBE/FDB 


|CBA/FDi 


3.94  +-  (30  X .00831) 
3.69  + (30  X .00831) 
5.79  + (30  X .00831) 


4.29  + (30  X .00857) 
4.07  + (30  X .00857) 
5.78  + (30  X .00857) 


CBA/TAE 

CBE/FDB 


modes  for  both  data  link  and  voice  link  messages  at  the  three 
traffic  mixes  at  full  load.  The  mode  ranking  by  shortest  CID  in 
Table  20  is  identical  with  the  mode  ranking  by  shortest  MTT  in 
Table  18,  indicating  quite  clearly  that  CID  is  the  governing 
component.  It  is  obvious  from  Table  19  that  CBE/FDB  is  at  a 
distinct  disadvantage  in  CID  with  respect  to  the  other  control 
modes  because  of  the  nominal  6 second  value  used  for  the  command 
display  time  parameter;  i.e.,  the  command  could  not  be  auto- 
matically dispatched  until  the  command  had  been  displayed  for 
approximately  6 seconds.  The  intent  in  using  this  parameter 
value  in  CBE  was  to  give  the  controller  time  to  not  only  evaluate 
the  command  but  also  to  disapprove  it  prior  to  automatic  dispatch 
However,  the  test  data  show  that  the  effect  of  the  6-second  CID 
was  to  bias  CBE/FDB  with  respect  to  CBA/FDB  and  CBA/TAB,  since 
in  the  latter  two  modes,  the  controller  was  not  constrained  in 
the  initiation  or  the  timing  of  Dispatch  actions.  Therefore,  the 
third  place  ranking  of  CBE/FDB  cannot  be  accepted  as  valid  or 


5.83  + (30  -X  .00831)  = 6.08 

4.33  + (30  X .00857)  = 4.59 

2.01  + (30  X .00831)  = 2.26 

2.34  + (30  X .00857)  = 2.6 

5.81  + (30  X .00831)  = 6.06 

5.78  + (30  X .00857)  = 6.03 

TABLE  20.  MODE  RANKING  BY  SHORTEST  CID 


MODE  RANKING 
DATA  LINK  CID 

MODE  RANKING 
VOICE  LINK  CID 

MODE 

20% 

50% 

80% 

20% 

50% 

80% 

CBA/FDB 

2 

3 

2 

2 

2 

2 

CBA/TAB 

1 

1 

1 

1 

1 

1 

CBE/FDB 

3 

2 

0 

3 

3 

3 
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conclusive;  its  ranking  is  totally  dependent  on  the  display-time 
parameter,  which  many  subjects  felt  could  be  reduced  (Post- 
Experiment  Question  C-14)  . 


The  ranking  of  CBA/TAB  ahead  of  CBA/FDB  is  consistent  with 
the  rankings  derived  from  the  results  of  the  subjective  data 
analysis.  Quite  obviously,  the  higher  MTT  values  for  CBA/FDB 
are  attributable  to  the  greater  CID  values  arising  from  the 
trackball  slewing  required  by  this  mode.  In  the  CBA/TAB  mode, 
trackball  slewing  was  not  required.  Although  the  controller  had 
to  correlate  the  topmost  blinking  command  in  the  list  with  its 
associated  blinking  aircraft  symbol,  commands  were  dispatched  by 
simply  depressing  a single  key. 


There  are  variations  in  the  MTT  of  the  candidate  modes  with 
the  three  different  percentages  of  data  link  aircraft  employed  in 
the  tests.  The  data  link  and  voice  link  MTT  values  for  maximum 
load  presented  in  Table  17  are  plotted  in  Figures  16  and  17  to 
illustrate  the  changes  that  occur  with  different  traffic  mixes. 
CID  values  associated  with  each  of  the  three  modes  are  also  shown 
in  Figures  16  and  17. 
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Figure  10.  Voice  Link  MTT  and  CID  at  Full  Load  - 60  Minutes 
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Figure  17.  Data  Link  MTT  and  CID  at  Full  Load  — 60  Minutes 
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The  questions  raised  by  Figures  16  and  17  are  [1)  why  CBA/FDB 
MTT  values  increase  at  501  before  decreasing  to  an  expected  low  at 
80%  and  (2)  why  CBA/TAB  MTT  values  act  conversely.  In  the  CBA/FDB 
case,  the  increase  at  50%  of  approximately  1 second  in  MTT  over  the 
20%  value  may  be  attributable  to  the  controllers  having  to  spend  a 
bit  more  time  in  seeking  out  a larger  number  of  response  indicators 
while  being  distracted  by  still  having  a reasonably  large  number  of 
voice  link  commands  and  pilot  acknowledgements  to  listen  to.  In 
the  CBA/TAB  case,  the  slight  decrease  from  20%  to  50%  is  probably 
because,  following  dispatch  and  VRS  enunciation,  the  controllers 
had  fewer  voice  link  commands  to  erase  from  the  tabular  list.  The 
command  erase  action  was  an  MSS  keyboard  function  used  by  the 
controllers  to  abbreviate  the  terminate  time  of  a voice  link 
command  which  had  been  dispatched  and  acknowledged.  The  terminate 
time  parameter  value  for  voice  and  data  link  commands  had  been 
set  to  20  seconds  for  the  tests,  so  that  in  cases  of  off-normal 
responses  (U,  F,  or  X)  data  link  commands  would  be  retained  for  a 
period  of  time  long  enough  to  allow  the  controller  readily  to 
correlate  them  with  the  response  indicator.  Unfortunately,  there 
were  no  provisions  for  setting  different  terminate  time  values 
for  voice  and  data  link  commands.  Therefore,  while  dispatched 
data  link  commands  were  automatically  erased  upon  receipt  of  a 
W,  dispatched  voice  link  commands  would  not  be  automatically 
erased  until  20  seconds  had  elapsed,  regardless  of  earlier 
acknowledgement  by  the  pilot.  Thus,  for  voice  link  commands,  the 
controllers  found  it  expedient  and  convenient  to  use  the  erase 
function  to  clear  each  acknowleged  voice  link  command  from  the 
tabular  list.  This  additional  workload  item  is  reflected  in  the 
slightly  longer  MTT,  20%. 

The  increase  in  MTT  of  approximately  2 seconds  from  50%  to 
80%  for  CBA/TAB  is  ascribed  to  the  additional  visual  workload 
imposed  on  the  controller  by  his  having  repeatedly  to  glance 
back  and  forth  from  the  tabular  list  to  the  ACID  line  of  the  data 
block  to  observe  the  incidence  of  the  W response  indicator  for 
the  much  larger  number  of  dispatched  data  link  commands. 


item  . 
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It  should  be  noted  that  MTT  at  801  for  CBA/TAB  and  CBA/FDB 
are  very  comparable,  indicating  that  the  disadvantage  of  trackball 
slewing  is  far  outweighed  by  the  advantage  of  having  a more 
integrated  command/response  display  location.  A control  mode 
combining  the  advantages  of  CBA/TAB  with  CBA/FDB  would  seemingly 
be  an  ideal  solution:  that  is,  a CBA/FDB  mode  in  which  commands 

are  displayed  in  the  third  line  of  only  one  data  block  at  a time 
and  dispatched  by  a single  key  depression  before  subsequent  com- 
mands in  another  data  block  can  be  displayed  and  dispatched. 

The  variations  in  the  overall  MTT  (data  link  and  voice  link) 
with  percentage  of  data  link  aircraft  for  each  mode  were  obtained 
by  averaging  the  data  link  and  voice  link  MTT  values  in  the 
proportions  indicated  by  the  mixes.  The  performance  of  the 
three  modes,  as  depicted  in  Figure  18,  again  compares  favorably 
with  the  rankings  based  on  controller  opinion,  especially  if 
one  recognizes,  as  previously  stated,  that  CBE/FDB  was  biased 
out  of  a possibly  higher  ranking  in  MTT. 

As  one  would  expect,  there  is  a correlation  between  message 
transaction  time  and  average  queue  size.  Indeed,  the  longer  it 
takes  the  controller  to  process  a command,  the  greater  the 
probability  that  there  will  be  an  increase  in  the  queue.  Table 
21  clearly  indicates  that  mode  rankings  on  the  basis  of  in- 
creasing average  queue  size  are  consistent  with  rankings  on  the 
basis  of  increasing  MTT.  Only  a single  set  of  queue  size  values 
is  shown  in  Table  21,  since  queue  size  does  not  change  signifi- 
cantly as  a function  of  traffic  mix. 


TABLE  21.  MODE  RANKING  BY  AVERAGE  QUEUE  SIZE 


RANK 

MODE 

AVERAGE  QUEUE 
SIZE 

1 

CBA/TAB 

0.238 

2 

CBA/FDB 

0.394 

3 

CBE/FDB 

0.457 
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Figure  18.  Combined  Data  Link  and  Voice  Link  MTT  and  CID 
at  Full  Load  — 60  Minutes 
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There  is  also  a high  degree  of  correlation  between  gate 
arrival  error  and  dispatch  of  the  final  heading  command.  Modal 
values  for  arrival  time  errors  at  the  gate  for  all  three  traffic 
mixes  are  shown  in  Table  22.  Since  they  are  calculated  by  sub- 
tracting the  scheduled  arrival  time  from  the  actual  arrival  time, 
positive  values  indicate  late  arrival.  The  magnitude  of  error  is 
a function  of  the  timeliness  of  dispatch  of  the  final  heading 
command  to  intercept  the  localizer.  Indeed,  in  the  CBA  modes, 
the  more  rapidly  the  controller  dispatches  the  final  heading 
command,  the  more  quickly  the  aircraft  executes  the  command,  and 
the  smaller  the  late  arrival  error.  The  difference  of  almost  1 
second  between  CBA/TAB  and  CBA/FDB  could  be  attributable  to  the 
trackball  slewing  required  in  the  latter  mode.  The  much  longer 
late  arrival  error  in  CBE/TAB  is  probably  due  to  the  fixed  com- 
mand initiation  delay  of  6 seconds.  The  absolute  magnitude  of 
these  errors  are  of  minor  concern  because  they  can  be  virtually 
eliminated  by  dynamically  adjusting  the  display  and  issue  time 
parameters  of  the  final  heading  command. 


TABLE  22.  GATE  ARRIVAL  TIME  ERRORS 


MODE 

AVERAGE  GATE 
ARRIVAL  TIME 
ERROR 

CBA/TAB 

0.55 

CBA/FDB 

1.46 

CBE/FDB 

6.27 

Of  greater  significance  than  gate  arrival  time  errors  is  the 
dispersion  of  arrival  time  errors  about  their  mean,  since  this  is 
a measure  of  the  degree  of  randomness  in  command  processing  time. 
Dispersion  in  the  delivery  and  execution  of  commands  contributes 
directly  to  dispersion  in  arrival  time  at  the  runway  threshold, 
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a key  factor  in  determining  runway  capacity.  In  fact,  the  errors 
in  timing  of  the  final  heading  command  are  effectively  doubled  at 
the  runway  threshold  because  of  the  trombone  approach  geometry. 

The  standard  deviation  of  arrival  time  error  at  the  gate  was 
6.2  seconds,  independent  of  mode  and  traffic  mix. 

The  conflict  data  collected  during  the  test  runs  were 
separated  into  three  categories:  (1)  detected  conflict  situations 

which  did  not  materialize,  (2)  actual  conflicts  not  detected,  and 
(3)  conflicts  detected  more  than  15  seconds  after  they  started. 

Of  greatest  concern  were  the  undetected  conflicts  (category  2), 
shown  by  mode  and  percentage  in  Table  23.  The  figures  in 
parentheses  represent  to  what  extent  the  conflict  count  could 
be  reduced  if  minor  conflicts  lasting  for  only  one  scan  were 
subtracted  from  the  associated  total  figures.  While  the  results 
of  the  statistical  analysis  are  not  statistically  significant,  the 
reduced  conflicts  counts  totaled  by  mode  tend  to  follow  the 
familiar  modal  ranking  pattern. 

TABLE  23.  UNDETECTED  CONFLICTS 


MODE 

201 

50% 

80% 

TOTAL 

CBA/FDB 

2(2) 

4(1) 

6(3) 

12(6) 

CBA/TAB 

5(1) 

2 

2 

9(5) 

CBE/FDB 

2(1) 

2(1) 

0 

4(2) 

In  comparing  the  conflict  data  obtained  with  the  data  link 
modes  against  that  obtained  with  the  voice-only  modes  (see  Table 
12),  the  only  meaningful  comparison  is  one  based  on  the  average 
number  of  serious  conflicts  (more  than  one  scan  duration)  which 
occurred  per  run  in  the  801  and  voice-only  modes.  The  figures 
shown  in  Table  24  are  the  average  number  of  conflicts  per  run 
which  occurred  over  six  voice-only  runs  and  six  801  data  link 


114 


,1.  ■ ) : __ 


TABLE  24.  AVERAGE  NUMBER  OF  CONFLICTS  PER  RUN  - VOICE-ONLY 
VERSUS  DATA  LINK 


MODE 

VOICE-ONLY 

80%  DATA  LINK 

CBA/FDB 

. 17 

. 5 

CBA/TAB 

.42 

.33 

CBE/FDB 

— 

0 

runs  in  each  mode.  While  neither  statistically  significant  nor 
conclusive,  these  figures  suggest  that  there  is  no  disadvantage 
to  data  link  operations  in  detecting  conflicts. 


11.  SUMMARY  OF  RESULTS 

11.1  MODE  RANKINGS 

Analysis  of  the  subjective  data  derived  from  controller 
questionnaires  shows  very  definitly  that  the  data  link  control 
modes  ranked  as  follows  in  order  of  preference: 

1.  Control-by-Exception/Full  Data  Block  (CBE/FDB) 

2.  Control-by-Approval/Tabular  List  (CBA/TAB) 

3.  Control-by-Approval/Full  Data  Block  (CBA/FDB) 

CBE/FDB  had  the  longest  message  transaction  time  (MTT)  of  the 
three  modes,  but  was  ranked  first  by  the  controllers,  despite  some 
concern  for  blunder  potential,  because  it  possessed  the  best  work- 
load, capacity,  and  stress  characteristics.  Longer  MTT ' s are 
inherent  in  CBE/FDB  because  of  the  fixed  command  disapproval  time 
which  in  effect  "biases"  this  mode  in  Command  Initiation  Delay  (CID), 
the  key  component  of  MTT.  CBA/TAB  had  the  shortest  MTT  but  was 
ranked  second  by  the  controllers  because  they  felt  that  operating 
from  a command  list  tended  to  divide  their  attention,  adversely 
affecting  workload  and  capacity.  The  low  MTT's  with  CBA/TAB  are 
attributable  to  the  single  key  dispatch  without  trackball  slew. 
CBA/FDB  had  the  second  best  MTT  but  was  consistently  rated  as  un- 
favorable in  workload,  capacity,  stress,  and  blunder  potential  be- 
cause of  the  necessity  for  continuous  trackball  slewing  actions. 

It  is  not  possible  to  rank  conclusively  all  of  the  modes  on 
the  basis  of  the  objective  data  measures  (MTT,  CID,  etc.)  because 
of  the  bias  factor  of  command  disapproval  time  in  CBE/FDB.  CBE 
will  always  have  a longer  CID  and  MTT  than  the  other  two  modes. 

It  is  noteworthy,  however,  that  CBA/TAB  and  CBA/FDB  can  be  ranked 
with  respect  to  each  other  and  that  this  ranking  is  consistent  with 
subjective  data  rankings  - i.e.,  TAB  ranked  above  FDB. 

11.2  CONTROL  AND  DISPLAY  PREFERENCES 

In  looking  more  closely  at  the  mode  rankings  on  the  basis  of 
controller  preference,  some  conclusions  can  be  drawn  as  to  the 
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display  and  keyboard  irput  aspects  of  each  mode.  CBE/FDB  ranks 
first  because  the  controllers  prefer  display  of  the  command  in  the 
data  block  with  which  they  constantly  deal  and  are  comfortable  in 
using  in  basic  ARTS.  The  third-place  ranking  of  CBA/FDB  does  not 
contradict  this  conclusion  because  CBA/FDB  entails  use  of  the 
trackball  which  is  universally  disliked  by  controllers.  The  single 
button  nontrackball  dispatch  scheme  of  CBA/TAB  is  obviously  the 
principal  reason  why  this  mode  ranked  second  despite  controller 
dislike  for  a command  list.  Another  reason  for  CBA/TAB's  higher 
ranking  may  be  a veiled  preference  for  single  command  over  multiple 
commands.  The  inference  that  can  be  drawn  here  is  that  a potentially 
desirable  CBA  mode  would  be  one  in  which  the  commands  were  displayed 
one  at  a time  in  only  one  data  block  at  a time  with  a single  button 
nontrackball  dispatch  action.  These  findings  are  in  substantial 
agreement  with  the  results  of  the  Phase  2A  experiment. 

11.3  CPERATIONAL  BENEFITS  OF  DATA  LINK 

Before  the  experiment  began,  most  of  the  controllers  felt  that 
MfjS  via  data  link  would  be  accepted  in  actual  operation  and  would 
favorably  affect  the  current  operational  system  in  an  all  data  link 
environment.  At  the  conclusion  of  the  experiment  the  attitude  was 
negative  due  primarily  to  the  undesirable  features  of  CBA/TAB  and 
CBA/FDB.  To  answer  the  same  question  on  the  basis  of  objective 
data  necessitated  extrapolation  of  80%  data  link  data  to  100%  and 
comparison  of  the  latter  with  data  obtained  from  non-data  link  voice 
only  runs.  The  nonlinearity  of  the  data  across  the  20,  50,  and  80 
percentages  indicated  that  a projection  to  100%  would  be  unreliable. 
The  very  limited  data  obtained  from  the  test  runs  involving  the 
voice-only  TAB  and  FDB  mode  of  operation  show  an  MTT  of  6.66  seconds 
for  both  modes.  In  c he  absence  of  data  on  a full  data  link  environ 
ment,  the  only  comparison,  albeit  a gross  one,  that  can  be  made  is 
to  compare  these  MTT's  with  those  obtained  at  80%  for  the  same  two 
modes.  In  data  link  TAB,  MTT  is  8.29;  in  FDB  it  is  8.06.  While 
these  figures  are  significantly  larger,  seemingly  suggesting  that 
in  terms  of  communications  delays  voice  link  is  superior  to  data 
link,  they  may  be  attributable  to  greater  familiarity  with  voice- 
only  operations. 
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11.4  OTHER  INFORMATION 


Other  results  of  this  simulation  study  indicate  the  following: 

1.  The  amount  of  additional  training  the  controllers  felt 
would  be  necessary  to  make  them  feel  comfortable  with  the 
operational  procedures  associated  with  a control  mode 
paralleled  the  mode  rankings  — i.e.,  1 hour  for 
CBE/FDB,  more  than  2 for  CBA/TAB,  and  more  than  4 for 
CBA/FDB. 

2.  Display  of  the  W character  and  simultaneous  erasure  of  a 
data  link  command  to  indicate  pilot  compliance  is  not 
necessary;  simple  erasure  of  the  command  would  be  suffi- 
cient. A majority  of  the  controllers  felt  that  the  3- 
second  display  time  of  the  W character  should  be  reduced. 

3.  In  the  control -by-approval  modes  the  commands  should  not 
stop  blinking  (timeout)  at  issue  time  but  should  remain 
blinking  and  be  eligible  for  dispatch  until  the  controller 
can  service  them.  This  would  require  that  the  algorithm 
be  capable  of  dynamically  updating  command  values  to  com- 
pensate for  delays  in  dispatching  what  would  otherwise  be 
obsolete  commands. 

4.  Voice  commands  should  be  held  over  in  a nonblinking  state 
for  a period  of  less  than  20  seconds.  This  could  be 
achieved  by  retaining  the  nonblinking  command  for  a short 
time  after  the  VRS  had  enunciated  the  command.  The  post- 
VRS  retention  time  could  be  based  on  average  pilot  voiced 
acknowledge  time. 

5.  Controllers  felt  that  the  workload  at  peak  load  was  slightl 
below  their  maximum  capacity  for  CBE/FDB,  slightly  above 
their  capacity  for  CBA/TAB,  and  far  above  their  capacity 
for  CBA/FDB.  They  were  divided  as  to  their  ability  to 
maintain  a picture  of  the  traffic  situation  for  conflicts 
while  issuing  commands.  If  Mf)S  and  data  link  failed  at 
peak  load,  half  the  subjects  felt  they  could  handle  the 
same  level  of  traffic  indefinitely.  Only  10%  thought  the 
situation  would  be  hazardous. 


Most  of  the  controllers  had  no  difficulty  in  interpreting 
commands  in  the  FDB  modes  despite  the  fact  that  there 
was  increased  clutter  and  more  manual  tag  repositioning 
required.  However,  401  of  the  controllers  had  difficulty 
recognizing  exception  condition  indications  (U,F,X)  in 
CBA/FDB  and  CBE/FDB.  The  majority  did  not  feel  that  ex- 
ception cases  were  time-consuming. 

A majority  of  the  controllers  felt  that  priority  should 
be  given  to  commands  around  the  base  leg  in  the  CBA/TAB 
mode . 

Most  of  the  controllers  felt  that  VRS  enunciation  of  com- 
mands was  less  tiring  and  that  the  3-second  interval 
between  VRS  transmissions  was  acceptable,  but  half  of 
them  felt  that  VRS  halting  and  starting  procedures  needed 
improvement  in  CBE/FDB.  Controllers  were  particularly- 
bothered  by  the  VRS's  inability  to  repeat  commands  and  to 
a lesser  extent  by  the  slower  VRS  talking  rate. 

Half  the  controllers  were  bothered  by  not  having  pretrans- 
mission approval  of  issued  commands  in  CBE/FDB.  On  the 
other  hand,  more  than  half  felt  6 seconds  was  sufficient 
for  disapproving  a command;  over  one  third  thought  it  too 
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12.  CONCLUSIONS 


in?  objective  of  this  study  was  to  ascertain  which  of  three 
candidate  control  inodes  was  the  most  acceptable  for  the  display 
and  dispatch  of  metering  and  spacing  commands  in  a mixed  voice/ 
digital  communications  environment.  The  conclusions  that  can  be 
reached  from  the  results  of  this  study  are  as  follows: 

1.  The  CBE/FDB  mode  is  the  most  favored  by  the  controllers 
despite  blunder  potential,  but  it  has  some  inherent  limi- 
tations that  result  in  longer  message  transaction  times. 

2.  As  was  concluded  in  the  Phase  2A  experiment,  a CBA/FDB 
mode  in  which  a single  command  at  a time  is  displayed 

in  one  data  block  for  dispatch  with  only  a single  keyboard 
entry  is  again  indicated  as  a potentially  viable  mode  in 
a future  system.  This  mode  would  combine  the  advantages 
of  both  the  CBA/FDB  and  CBA/TAB  modes  as  currently 
implemented . 

3.  Contrary  to  the  findings  of  the  Phase  2A  experiment,  com- 
puter-generated voice  was  not  favorably  regarded  because 
of  the  talking  rate  and  inability  to  repeat  commands. 


13,  RECOMMENDATIONS 


1 


Further  studies  aimed  at  determining  the  optimum  means  for 
interfacing  the  controller  and  the  computer  in  a data  link  com- 
munications environment  should  be  conducted  in  the  following  areas: 

1.  Additional  testing  of  the  Control -by-Except ion/Ful 1 Data 
Block  mode  of  operation  should  be  conducted  with  an  on- 
line controller-selectable  disapproval  time.  This  will 
allow  a controller  to  decrease  MTT  if  he  feels  disapproval 
time  need  not  be  as  long  (6  seconds)  as  it  was  in  this 
study . 

2.  On  the  basis  of  the  findings  of  both  Controller/Computer 
Interface  experiments  (Phase  2A  and  2B) , it  is  recommended 
that  a CBA/FDB  mode  in  which  a single  command  is  displayed 
at  one  time  in  an  aircraft  data  block  for  dispatch  with 
only  a single  keyboard  entry  be  incorporated  in  a future 
system. 

3.  Considering  the  advantage  of  computer- generated  voice  in 
providing  commonality  of  command  dispatch  in  a mixed  voice/ 
digital  communications  environment,  it  is  recommended  that 
future  research  be  devoted  to  improving  the  talking  rate 
and  in  developing  a command  repeat  capability. 
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APPENDIX 

TIMELINES  FOR  EACH  CONTROL  MODE 


[ 


The  sequence  of  events  occuring  in  each  of  the  control  modes 
evaluated  in  the  test  series  are  graphically  described  in  the  time 
lines,  Figures  A-l  through  A-ll.  Each  of  the  three  data  link  con- 
trol modes  has  separate  time  lines  for  data  link  and  voice  link 
aircraft.  The  sequence  of  actions  by  the  computer,  the  controller 
the  voice  response  system,  and  the  pilot  is  indicated  by  coded 
markers  on  the  timeline.  Certain  intervals  of  special  importance 
in  evaluating  the  formats  are  defined  on  each  timeline.  These 
timelines  are  referred  to  in  Section  5,  "Test  Conduct."  The 
symbols  used  in  the  timelines  are  as  follows: 

^ = computer  action 
A = controller  action 
A = simulator  pilot  action 
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S PATCH 


Figure  A-l.  CBA/Tabular  List/VRS-Data  Link  Aircraft 
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Figure  A-3.  CBA/Full  Data  Block/VRS-Data  Link  Aircraft 


Figure  A-4.  CBA/Full  Data  Block/VRS-Voice  Link  Aircraft 
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Figure  A-6.  CBE/Full  Data  Block/VRS-Voice  Link  Aircraft 


Figure  A-7.  CBA/TAB/VRS-Data  Link  Aircraft  — Unable 
or  Fail  Exception  Case 


Figure  A-9.  CBA/Full  Data  Block/VRS-Data  Link  Aircraft 
Unable,  Fail,  and  Timeout  Exception  Cases 
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Figure  A-ll.  CBA/Full  Data  Block/Voice-Voice  Link  Aircraft 


